DBT as the “Couch” for Organizational Therapy
May 13, 2021 § 1 Comment
Or, “How ELTT is the Key to World Peace”
Draft Submission Script for Coalesce 2021
Hey there Data Lovers, my name is Dr. Ernie. And this is my English Cocker Spaniel Qhuinn, who with me and my boss make up the IT department at a Palo Alto startup.
I am Caltech physicist turned management consultant turned Apple Product Manager. During my 17 years there during the second coming of Steve Jobs, I pioneered Apple’s efforts in Open Source and server software. Which is arguably why people like you now develop on sexy Mac laptops that play nicely with your enterprise data systems.
You’re welcome.
Seven years ago I walked away from all that success to become a struggling entrepreneur, which led to several family members seriously questioning my sanity! After four counselors, three psychiatrists, two psychologists — and maybe a partridge in a pear tree — I discovered the core skill I needed to cultivate is marrying self-differentiation with other-integration. Put another way: I needed to learn how be authentically myself in a way that also empowers others (and even myself) to authentically be in community with me.
Modern enterprise data is facing precisely that same existential crisis. Originally all data was owned and operated purely by central IT. It was often wrong and inevitably myopic, but at least it was consistent.
Over successive tech generations, individual stakeholders and departments carved out greater control of their own data. Today, turnkey SaaS solutions mean edge data stores are as (if not more!) powerful than central repositories.
The tragedy is that these local optimizations result in global chaos. Each team gets religiously attached to their view of the data, and frustrated that others don’t work with their operational reality. We move faster within our own islands, but slowly or even backwards as a corporate entry.
#Election2016
- Extract
- Load
- Transform
- Test
The beauty of “load-first” architectures is that they accept every producer’s data “just as they are.” Of course, this is also why data lakes turn into data swamps. The beauty of DBT is that it allows us to apply modern software engineering practices to the “transform and test” portions, so we can give each consumer precisely the data they need.
At least in theory.
In practice, more powerful tools and shorter cycle times mean that we “get to” spend less time on coding SQL and “have to” spend more time understanding the deeper needs, motivations, and pain points of stakeholders. My thesis is that, in order to fulfill the bold promises we are making, analytics engineers will need to become “organizational therapists” who can embrace and reconcile the fragmented and contradictory aspects of our corporate personas.
This is not easy, and probably wasn’t what any of us signed up for. But if COVID-19 taught us nothing else, it is that we are more connected — and more broken — than we ever imagined. Yes, this is a call to step up and learn the “real skills” we need to properly do our jobs, empower our stakeholders, and accelerate our businesses. But more than that, this is a fantastic excuse to do the personal development that makes us — and the institutions we belong to and care about — more humane, empathic, inclusive, and effective.
Who’s with me?
iHack, therefore iBlog 
[…] Based on “DBT as Organizational Therapy“ […]