How to Build LightDash from Source
November 11, 2021 § Leave a comment
LightDash is a super-cool Open Source business intelligence tool built on top of DBT (which I think of as node for SQL). While it is distributed as open source, the usual way to deploy it locally is by simply running a docker container.
If you want to actually built lightdash directly from source yourself, you need to follow the instructions under CONTRIBUTING. However, what was written there (as of November 11, 2021) did not quite work for me, so here are my workarounds.
I will also file this as a GitHub issue, and they are super-responsive so hopefully this page will be obsolete soon!
« Read the rest of this entry »Configuring DataBricks on AWS
May 5, 2021 § Leave a comment
Despite the excellent QuickStart tools, this was way harder than I thought. For some reason I had the worst difficulty creating a Workspace on AWS for Databricks.
Here are some tips that might help others who get stuck.
« Read the rest of this entry »SyncHouse: MVC for Enterprise SaaS
May 2, 2021 § 1 Comment
A concrete proposal for Imagining a Data Resort as enforcing a Model-View-Controller architecture across multiple Software-as-a-Service applications. The key is replacing transient enterprise data integrations with a persistent “sync house,” and making that the one full-service Source of Truth for data, schemas, and business logic.
- Ingest data from Salesforce, NetSuite, etc. (e.g.,
Stitch/Talend, FiveTran) - Store raw data in a LakeHouse (e.g., Databricks, Delta Lake; or just Redshift)
- Aka “ELT vs ETL“
- Manage schemas via dbt (e.g., dbt Cloud)
- View and report on appropriate data (e.g., Mode, Data Studio)
- Push updates (reverse ETL) back to source applications (e.g.,
Celigo, Get Census)
Imagining a Data Resort
April 21, 2021 § 1 Comment
A data resort is where data comes to get pampered, so that it is prepared to get back to work.
Motivation
The good news is that I finally understand how we really need to be managing all the business data in my organization. The bad news is that I don’t know how to articulate that in terms of industry-standard terminology (examples below). Worse, I’d probably use the wrong term (or the right term incorrectly) leading to endless rounds of frustrating conversation.
Therefore I’ve coined a new term (“data resort“) which I can define as the exact thing I want. Hopefully you my dear readers can help translate that into something concrete I can efficiently buy+build today!
Let me know your suggestions in the comments or via email.
« Read the rest of this entry »My First Date with Quilt Data
July 21, 2020 § Leave a comment
I’ve known the good folks at Quilt Data for a long time. A company hackathon gave me a good excuse to actually use them “in anger” for an actual demo. These are my notes on how to configure quilt3 and create my first package (and panda data frame) from a CSV
« Read the rest of this entry »SSO Login into Salesforce from Node via samlp SAML IdP
October 4, 2019 § Leave a comment
Documenting this in a blog post because it drove us crazy trying to figure out exactly what was involved, even though it was actually easy to implement once we understood all the terminology.
In order for our previously-authenticated users to automatically log into Salesforce, we needed to:
- Create a “/sso-url” on our node server for our web app to access
- When our web app GETs that URL, create and a return a SAML Identity Provider (IdP) using
samlp - That IdP is interpreted by the web browser a redirect to the Salesforce URL (returned by the function assigned to `getPostURL`)
- Salesforce just needs to have the IdP certificate and Entity ID in its SSO Settings
Below are additional details on why we needed this.
TypeScript 2: Adding custom.d.ts Typings Files for Existing JavaScript Modules
March 22, 2017 § Leave a comment
If you are trying to import ordinary JavaScript modules into a TypeScript project, and those modules don’t already have an internal or external typings file, you may get an error like:
- error TS2688: Cannot find type definition file for ‘lodash’
- error TS7016: Could not find a declaration file for module ‘himalaya’
For some reason, the TypeScript documentation talks a lot about how to create typings files for modules that don’t have them, but not about how to add them to your project. And it is incredibly difficult to Google the correct answer.
The best answer I found is from: https://www.bennadel.com/blog/3169-adding-custom-typings-files-d-ts-in-an-angular-2-typescript-application.htm
The actual answer is quite simple: are only four steps:
$ npm install -g --save typings # if you haven't already $ cat > custom.d.ts # type definition file for that module declare module 'custom'; ^D $ typings install --global --save file:custom.d.ts
That’s it!
Auto-launching Chrome to Debug Mocha Tests on Mac
March 22, 2017 § 1 Comment
Are you tired of cutting and pasting URLs from node –inspect to launch Chrome?
Stark Realty #20: Redemption
March 27, 2016 § 1 Comment
[The following is the final installment of a science-fiction serial I started writing at Matterport, where I worked from May to December of 2015. Someday I will get the rights to publish the entire story, but for now, enjoy this little vignette.]
Stark Realty #20: Redemption
“He didn’t kill your mother, Tanya.” panted the newly-arrived Jane Hathaway. “I did.”
Tanya Bain stared at the petite, matronly woman in shocked disbelief. Tony Stark, his hands nailed to the concrete wall with Q-carbon spikes, screamed “Jane, no!”
The demolecularizing grenade Tanya had placed over her father’s arc reactor-powered heart dropped from suddenly nerveless fingers. With a supreme effort, Tony levered himself against the spikes and drop-kicked the grenade against the far wall, where it harmlessly vaporized a quarter-inch of foamed concrete.
“Tanya,” he gasped. “Please. Don’t believe her. It was all my fault.”
His estranged daughter stared at him blankly, sinking slowly to the ground. She, with help from the anonymous cyber-dwarf Rumplestilskin, had put herself through hell to destroy Iron Man for killing her mother. Had her whole life been based on a lie?
iHack, therefore iBlog 
You must be logged in to post a comment.