SyncHouse: MVC for Enterprise SaaS
May 2, 2021 § Leave a comment
A concrete proposal for Imagining a Data Resort as enforcing a Model-View-Controller architecture across multiple Software-as-a-Service applications. The key is replacing transient enterprise data integrations with a persistent “sync house,” and making that the one full-service Source of Truth for data, schemas, and business logic.
- Ingest data from Salesforce, NetSuite, etc. (e.g.,
- Store raw data in a LakeHouse (e.g., Databricks, Delta Lake; or just Redshift)
- Aka “ELT vs ETL“
- Manage schemas via dbt (e.g., dbt Cloud)
- View and report on appropriate data (e.g., Mode, Data Studio)
- Push updates (reverse ETL) back to source applications (e.g.,
Celigo, Get Census)
More recently, though, it occurred to me that the deception went much deeper than that. The dominant SaaS platforms implicitly assume they are each the center of reality. While inter-application integration is easier than ever, the brittleness of the underlying model is a constant thorn in my side. Not only I am continually having to debug and fix broken workflows, but I also have to manually update production data to keep the exceptions under control. It feels like performing trapeze acts without a net — while the circus is on fire!
It suddenly occurred to that the whole reason I felt so trapped is because I had bought into the conceit that those systems were reality, and integrations merely a transient connection between them. What if we instead flipped the paradigm? What if we treated the integration point as reality, and demoted our SaaS applications to being mere UI views, data caches, and compute engines?
Might this allow us to bring the full power of modern software development practices and tooling to bear on enterprise data? Could I finally use best-of-breed programming environments to manage critical business logic (“an IDE for EDI”)? Could we build it all on an immutable data store so I never have to worry about irretrievably overwriting critical business information? Can we finally start to manage data like code?
In short, can we finally make IT earn respect (and compensation, and business impact) equivalent to software engineers?
Stay tuned to find out…