SyncHouse: MVC for Enterprise SaaS
May 2, 2021 § 1 Comment
A concrete proposal for Imagining a Data Resort as enforcing a Model-View-Controller architecture across multiple Software-as-a-Service applications. The key is replacing transient enterprise data integrations with a persistent “sync house,” and making that the one full-service Source of Truth for data, schemas, and business logic.
- Ingest data from Salesforce, NetSuite, etc. (e.g.,
Stitch/Talend, FiveTran) - Store raw data in a LakeHouse (e.g., Databricks, Delta Lake; or just Redshift)
- Aka “ELT vs ETL“
- Manage schemas via dbt (e.g., dbt Cloud)
- View and report on appropriate data (e.g., Mode, Data Studio)
- Push updates (reverse ETL) back to source applications (e.g.,
Celigo, Get Census)
Backstory
I accidentally ended up in Enterprise IT when the startup I joined realized they needed administrative help more than they needed a novice GoLang Engineer. A first I was relatively pleased that at the agility and flexibility of modern SaaS applications, which I could customize with “near-standard” versions of JavaScript and Java. However, I quickly realized that the bane of my existence would be programming in non-programming environments, without the benefit of the rich IDEs, testing, and version control systems I had come to expect.
More recently, though, it occurred to me that the deception went much deeper than that. The dominant SaaS platforms implicitly assume they are each the center of reality. While inter-application integration is easier than ever, the brittleness of the underlying model is a constant thorn in my side. Not only I am continually having to debug and fix broken workflows, but I also have to manually update production data to keep the exceptions under control. It feels like performing trapeze acts without a net — while the circus is on fire!
It suddenly occurred to me that the whole reason I felt so trapped is because I had bought into the conceit that those systems were reality, and integrations merely a transient connection between them. What if we instead flipped the paradigm? What if we treated the integration point as reality, and demoted our SaaS applications to being mere UI views, data caches, and compute engines?
Might this allow us to bring the full power of modern software development practices and tooling to bear on enterprise data? Could I finally use best-of-breed programming environments to manage critical business logic (“an IDE for EDI”)? Could we build it all on an immutable data store so I never have to worry about irretrievably overwriting critical business information? Can we finally start to manage data like code?
In short, can we finally make IT earn respect (and compensation, and business impact) equivalent to software engineers?
Stay tuned to find out…
iHack, therefore iBlog 
[…] we are contemplating moving to a SyncHouse with a The Reporting Control Center, my dream would be to go one step beyond “live […]