Benn’s “rant” feels profound on so many levels, especially if I can assume he’s captured the zeitgeist of our industry as accurately as he usually does.
My first observation is that he seem to (wisely!) invert Postel’s Law for data: be strict in what you accept, and generous in what you emit. The profound truth here is that we cannot control other people. We can only honestly and gracefully fail, if we are not getting what we need to succeed.
We can only honestly and gracefully fail, if we are not getting what we need to succeed.
I can’t help but wonder how much of the energy around “data contracts” is the desire to avoid facing exactly that reality.
Next, the corollary to this is something I literally wrote last night in an internal planning document: “transparency is more important than compliance”. The context is that don’t want employees worried about “appearing” to reach nominal goals. I want them to be ruthlessly honest with us about the true risks to delivering genuine impact.
“Transparency is more important than Compliance”
Third, the profound implications of this is that we must shift power from centralized hierarchies to decentralized networks. We have to stop chasing Xanadu — the mythical demo of reliable hyperlinks — and embrace the chaotic generativity of the World Wide Web. That is the only kind of system that ever truly scales.
Shift power from centralized hierarchies to decentralized networks
Finally, Benn is right that it is foolish to replace a technical problem with a human problem. But I fear you can never avoid the human problem, only squish it somewhere else. The challenge is finding the “right” human problem to solve, so the rest of the system can support that as efficiently as possible.
Finding the “right” human problem to solve, so the rest of the system can support that as efficiently as possible.
I think Benn is calling for pipelines to “fail quickly” when it is better for consumers to get explicitly old data versus implicitly wrong data. But that implies non-fatal errors must be communicated transparently yet efficiently throughout the stack.
This is literally impossible (née Masnick), but I believe it is THE human problem that must be addressed — even if we can never solve it! Once we embrace that ugly truth, we can devote all of our effort to doing the best we can technically, while giving each other grace to recognize our human limits.
That’s a contract I’m willing to sign up for. How about you?
A key design goal of PipeBook is to break away from the single-browser-window user experience of traditional data notebooks, to take full advantage of the large screens on today’s laptops and desktops.
The modern data notebook has its roots in academic tools for mathematical research. Because of that, notebooks are fantastic for open-ended exploration, but an awkward match for production data pipelines. In particular, they don’t:
Explicitly declare and track dependencies
Enforce organizational quality and reproducibility standards
Enable easy testing, validation, and alerting
PipeBooks are a simple but radical re-imagining of notebooks as “tools for iteratively constructing resilient data pipelines.” The key is a novel data format called FRIDAAY that allows us to:
Khommunity Enrichment Malls as the Pre-Future of School
As she walked up the corridor on her Last Day, Molly kept telling herself she shouldn’t be here.
No, whispered a cynical voice in her head. You shouldn’t even be alive.
Molly
She was born without legs, and only half an arm. When her parents died during the Anti-Race Riots of the Second Civil War, it seemed inevitable that she would soon follow them — and America — into oblivion.
As she gazed around the Mall that had become her sanctuary, she felt a crushing sadness that she had to leave. She had begged them, pleaded with them, demanded that they let her stay. At least a little longer.
But the Elders had been adamant. Given her capacity, and her ranking at the end of the scale, they had no choice. The world was a cold, unforgiving place; someone had to make the hard decisions, or none of them would survive.
Elder Wiggin, whom she had always thought kind, had put it the most bluntly: “If we let people like you stay here, we fail everyone else. Those Outside will think we are soft and lazy, and deserve to die. And they will be right.”
Those Outside will think we are soft and lazy, and deserve to die. And they will be right.
Still, she had to be honest: she had brought it on herself. Staying up all night reading subversive political literature. Spending class-time playing simulation games with strangers from all over the world. Arguing with her tutors. Writing detailed critiques of the elders, and how they ran the Mall.
It just seemed so unfair that they would move her Last Day up to November 7th. Why couldn’t they have at least let her stay until Season’s End, when the other “ejects” (as they called themselves) were allowed to leave together. Its not like she had anywhere urgent to go before then.
But the elders were unsympathetic. The final tally had come in. Her fate had been decided. There was no point in delaying the inevitable. Best for her to go, and let the rest of them move on with their lives.
The Mall
She cast one last wistful look at the storefront of the Catholic orphanage that had been her home for so many years. Poor in money but rich in love, they had somehow guilted the original developer of the Mall into giving them space, and nobody since had had the heart to kick them out.
The Mall. The original Khommunity Enrichment Mall. It started out as basically a warehouse for online learners, combining all-day childcare with a wide range of supervised and unsupervised after-school activities, from a diverse range of independent providers. An innocent-sounding idea for what was (in retrospect) an innocent time.
Over the years, the Mall grew: politically by incorporating as a city, physically by annexing territory, and pedagogically by becoming the first true hub for lifelong learning. Over time the ecosystem mix organically shifted from tutoring and day camps towards vocational training, digital media, light industry, and startup accelerators.
The first true hub for lifelong learning… the ecosystem mix organically shifted towards vocational training, digital media, light industry, and startup incubators
The nuns remembered the 2020s as some kind of Golden Age, though Molly knew from her reading that that generation was already drugged, divided, and distracted by the unfolding metacrisis. When that erupted into a full-blown meltdown, the Mall and its sisters became the last bastion of civilization, like Benedictine monasteries two millennia ago. They taught people how to maintain the power grids, network infrastructure, and agricultural supply chains that stitched together all that was left of a once-proud nation.
The Malls provided some measure of safety and stability in an increasingly lawless world. But at a heavy price.
The Discipline
To cope with overcrowding from the influx of refugees, the most powerful politicians rechristened themselves Elders, turned an educational monitoring system into The Discipline, and instituted the Life Thesis to determine who would go and who could stay.
The Discipline went by many names: Datocracy. Mastery Transcript. The Fishbowl. Big Brother. Basically, it amounted to a complete and total lack of privacy, with every online (and most offline) interactions recorded, uploaded, and indexed. Cryptographically secure and fully public, so — at least in theory — every Mall rat had the same information and access as an Elder. And nobody had anywhere to hide.
Sure, you always had the option to “go dark” by using an official (or unofficial) VPN, but the social pressure to stay online was intense. Just like villagers for most of human history, people only trust those they (think they) know everything about.
The Discipline provided the raw material for defending your Life Thesis: the story you told about your value to society. The Elders claimed it was about helping you find your proper place, but everyone knew the stakes were much, much higher: determining who would stay, and who would be sent Outside. On one side of the line were the strong, the talented, the brilliant, the savvy. The ones with the most potential to contribute. On the other side — everyone else.
Molly had, for obvious reasons, needed to delay her Life Thesis until her thirties. Two friends, Mano and Miriam, had — despite her urgings — used their Theses to create the exoskeleton she used to walk, and the prosthetic hand she used to type.
She still missed them.
The Thesis
Molly had been sorely tempted to play it safe. Tell the Elders a story that would inspire them to keep her there. Play on their sympathy. Misrepresent what she was capable of.
She tried to make herself fit in. She really tried. But she just couldn’t do it.
Instead, she let them have it. Told them what she really thought. Perhaps what everyone else thought, but was too afraid to say.
That the Mall system was headed for total and complete failure. Every year, the darkness Outside grew worse. No matter how fast we grew and built Malls, or invented new technologies, it couldn’t keep pace with the decay of the planet. Or of the people.
The only solution was tear down the walls and reintegrate the Malls back into the rest of society. Seek to renew civilization from within, rather than lecture it from outside. Yes, it was risky. Yes, it was dangerous. But better to take a gamble that might pay off than the sure bet of a slow suicide.
There. She had said it. There was no taking it back, even if she wanted to. Every word copied into every node of the Mall blockchain, recorded permanently for anyone who ever cared to look.
The Verdict
Of course the Elders reacted exactly as she had expected. And so had everyone else. Which is why this was her Last Day.
Molly reached the end of the aisle. Elder Wiggin stepped to the podium, her normally animated face immobile, like a judge about to pass sentence. Though something else seemed to lurk behind it.
“Molly, today is your Last Day. Though it has been long in coming, I for one cannot say I am surprised.”
Murmurs ran through the crowd, quickly hushed as Wiggin’s face shifted briefly into her famous frown.
“You came to us as a cripple, and a child. We fed you and clothed you, gave you unfettered access to all the world’s information. You worked with the latest technology, studied with the most brilliant minds, and were trusted with our most important problems.
“And how have you repaid us? By challenging the very institutions that kept you alive. Critiquing the Elders who sacrificially watched over. Hassling your teachers with impossible questions. Calling for the end of everything you have ever known… and loved.”
The mask cracked. Molly thought gentle Elder Wiggin might dissolve into tears at the enormity of what was to happen next. Molly herself wasn’t far behind.
Somehow the octogenarian Elder recovered enough composure to whisper into the microphone.
“And I… I couldn’t be more proud!”
The crowd, which had restrained itself admirably, burst into thunderous cheers. Wiggin didn’t even try to stop them, but waited until the sound tapered off.
“The Mall was never meant to be an Ivory Tower, but an incubator. We always knew that the very isolation and rigidity that allowed us to survive would eventually be our undoing. But we had no idea when we should make that transition, much less how. We dared not even speak of it outside ‘dark time’ lest it engender even more fear and insecurity in those already traumatized.
“We always knew that the very isolation and rigidity that allowed us to survive would eventually be our undoing.”
Elder Wiggin
“But then you came into my life. Dear, sweet, trouble-making Molly. You filled a hole in my heart after my children and grandchildren perished. And you were the final piece of the puzzle. As you grew in wisdom and maturity, my suspicion that you might be The One we were waiting for grew into certainty.
“My only fear was that those Outside might not welcome her radical new ideas as readily as we did. As you all know now, I need not have worried!”
A low chuckle runs through the crowd.
“It is always a proud yet painful Last Day when we send our best and brightest Outside into the world. But never so much as this one. Ladies and gentlemen, I give you my adopted daughter, Molly Wiggin: the next President of these United States!”
TL:DR Businesses may start by developing a technical solution, but only succeed by integrating around a human problem. The same is true of the Modern Data Stack.
The Human Colossus is about to undergo a phase transition (“from rational to relational“) analogous to the one where Gutenberg, Luther and Newton inaugurated modernity (“from ritual to rational“)
[These are my initial musings. It may take weeks or months to turn these into a coherent analysis, so I figured I should publish them as-is to get them out into the world. Merry Christmas!]
Challenge Question: What is the minimum number of bits necessary to meaningfully simulate some aspect of an emotion? « Read the rest of this entry »
iHack, therefore iBlog 
You must be logged in to post a comment.