Imagining a Data Resort

April 21, 2021 § 1 Comment

A data resort is where data comes to get pampered, so that it is prepared to get back to work.

Motivation

The good news is that I finally understand how we really need to be managing all the business data in my organization. The bad news is that I don’t know how to articulate that in terms of industry-standard terminology (examples below). Worse, I’d probably use the wrong term (or the right term incorrectly) leading to endless rounds of frustrating conversation.

Therefore I’ve coined a new term (“data resort“) which I can define as the exact thing I want. Hopefully you my dear readers can help translate that into something concrete I can efficiently buy+build today!

Let me know your suggestions in the comments or via email.

Requirements

  1. Data Lake: schema-on-read access to an immutable, append-only data store
  2. Connectors: easily import and export data via dynamic schemas over multiple protocols (eg, HTTP, SMTP) to REST, SalesForce, NetSuite, etc. Should use (or be as easy as) Celigo’s integrator.io.
  3. Logic Layer: git repository plus execution engine for queries, triggers, and automations using standard languages (e.g., JavaScript, SQL)
  4. Console: unified web interface for browsing, configuring, debugging — and ideally editing — queries, reports, data and logic
  5. Time Travel: trivial to run (and idempotently rerun) any version of logic against any snapshot of data, e.g., staging servers, sandboxes, 2019 data against 2021 logic, vice versa, etc.
  6. Access Control: fine-grained permissions of who can do what
  7. Auditing: fine-grained tracking of who did what when

My dream is that this would subsume (and eventually eliminate or at least deprecate):

  • Analytics Engines (Tableau, Qubole)
  • Data Warehouses
  • In-App Scripting and Alerts (Salesforce, NetSuite)
  • Manually-Prepared Audits
  • Mutable Databases for Business Data
  • Storage as a Service (Box, DropBox)
  • Sync Engines (Celigo, DELL Boomi)

Procurement

I would love to buy a turnkey solution for this from a top-tier vendor. Unfortunately, all the ones I know about have strong incentives to avoid empowering customers to this extent.

Intriguingly, I hear rumors that large enterprises “at scale” organically end up with an architecture like this, but by that point it is customized and idiosyncratic that it can’t be used by anyone else. My hope is that the various components have finally become mature enough a dedicated amateur can cobble something together at a low-enough cost for our medium-sized business.

Therefore, I suspect this will require some combination of:

  1. Best-of-breed commercial solutions for key components
  2. Open source modules for commodity functionality
  3. A hungry young startup eager to vertically integrate into a full-stack solution
  4. Custom development to fill in the gaps

Existing Data- Terminology

  1. Analytics
  2. Base
  3. Broker
  4. Catalog
  5. Center
  6. Connector
  7. Custodian
  8. Frame
  9. Governance
  10. Integration
  11. Integrity
  12. Lake
  13. Lab
  14. Library
  15. Loader
  16. Management
  17. Mart
  18. Mesh
  19. Mining
  20. Model
  21. Path
  22. Record
  23. Repository
  24. Schema
  25. Set
  26. Source
  27. Strategy
  28. Store
  29. Table
  30. Tree
  31. Verse
  32. View
  33. Warehouse

See Also

QuantOps 2030: Ten Years into the Analog Revolution

Tagged: , ,

§ One Response to Imagining a Data Resort

Leave a Reply to SyncHouse: MVC for Enterprise SaaS | iHack, therefore iBlog Cancel reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

What’s this?

You are currently reading Imagining a Data Resort at iHack, therefore iBlog.

meta

%d bloggers like this: