Trusted Data
Operations For
The AI Era

Uber built Apache Hudi to solve a familiar data problem: Records change after they land.

• A trip gets corrected after it ends.

• A chargeback arrives weeks later.

Rebuilding a large table to fix a handful of rows is wasteful.

Our free idempotent backfill skill applies some of Hudi's design ideas to late-arriving corrections in data you've already published. Give the current URL to your coding agent and simply tell it to install the skill to try it out.

What we borrowed from Hudi

Uber's lakehouse write-up describes a backfill that reads a fixed snapshot and overwrites only the affected partitions. It does not move the incremental writer's checkpoint. That separation prevents an old repair from disrupting the live pipeline.

Our skill turns a few of its core practices into a six-step, database-independent checklist:

1. Fix the source snapshot.

2. Name the partitions, business key, and newest-wins order.

3. Build each partition twice.

4. Compare the logical rows.

5. Stop for approval.

6. Replace and verify one partition at a time.

The skill is short on purpose so the agent can use the database's own atomic replacement operation.

Why build it twice

Maxime Beauchemin's essay on functional data engineering gives us a clear rule. Treat a partition as the complete output of a function. The same source snapshot and transform should return the same rows every time.

Every capable agent (human or artificial) needs two things before it can act with confidence: a clear picture of where we are, and a robust definition of where we need to go.

That sounds obvious. It is also where most agentic systems quietly fail.

We have poured enormous energy into making models smarter, tools more composable, and orchestration layers more sophisticated. Yet the hard problem is not reasoning in the abstract. It is grounding that reasoning in a faithful account of the present (here) and an unambiguous specification of the intended future (there). Without both, an agent is improvising. With both, it can plan, execute, verify, and explain.

Agents Don't Need Magic. They Need Context with Edges.

An AI agent assisting a mission, a business process, or a data pipeline is only as good as the situation it can see and the outcome it is asked to produce. "Current state" is not a chat transcript. "Goal" is not a vague aspiration. Both must be detailed enough that another competent actor (software or human) could inspect them, challenge them, and act on them.

That means capturing:

• Here: what exists now (systems, sources, constraints, policies, dependencies, quality, ownership, and known gaps). Not a slide. Not a tribal memory. Ground truth.

• There: what "done" looks like (required outcomes, acceptance criteria, interfaces, governance rules, and the boundaries the agent must not cross).

SysML Repo Modeler is now open source and free to use. It turns one GitHub repository—or many—into an explorable SysML v2 system model. Get the code on GitHub.

It gives us a coherent way to visualize our own repositories, services, APIs, and layers of institutional knowledge in an exploratory systems view. And now it’s open source for anyone to freely use.

Explore a Live Model

Want to see the result before getting into the details? Start with the live Supabase Platform model, which maps five repositories and four languages as one system. You can also explore the OpenClaw model, the n8n model and the Ollama model. All four sit side by side in the SysML Repo Modeler gallery, with the repository, part and connection counts for each.

Open the live Supabase Platform model — click the image to search, filter, and explore the system.

The Problem: Architecture Lives in Too Many Places

The reality is that modern systems are scattered across multiple repositories, services, APIs, and layers of institutional knowledge. The challenge is not that teams lack documentation; it is that documentation struggles to keep up with what the code does.

Traditional Model-Based Systems Engineering (MBSE) documentation can be useful, but it is often manually maintained across disparate software platforms that require significant user knowledge and training, with limited ability to transfer data between tools. As a system evolves, the diagram becomes a snapshot of what people thought a system looked like but not necessarily what exists now.