Uber built Apache Hudi to solve a familiar data problem: Records change after they land.
A trip gets corrected after it ends.
A chargeback arrives weeks later.
Rebuilding a large table to fix a handful of rows is wasteful.
Our free idempotent backfill skill applies some of Hudi's design ideas to late-arriving corrections in data you've already published. Give the current URL to your coding agent and simply tell it to install the skill to try it out.
What we borrowed from Hudi
Uber's lakehouse write-up describes a backfill that reads a fixed snapshot and overwrites only the affected partitions. It does not move the incremental writer's checkpoint. That separation prevents an old repair from disrupting the live pipeline.
Our skill turns a few of its core practices into a six-step, database-independent checklist:
Fix the source snapshot.
Name the partitions, business key, and newest-wins order.
Build each partition twice.
Compare the logical rows.
Stop for approval.
Replace and verify one partition at a time.
The skill is short on purpose so the agent can use the database's own atomic replacement operation.
Why build it twice
Maxime Beauchemin's essay on functional data engineering gives us a clear rule. Treat a partition as the complete output of a function. The same source snapshot and transform should return the same rows every time.
That is why the skill replaces the whole partition. An append keeps whatever a previous run left behind, so the output depends on prior state instead of the declared inputs.
What happens if this job runs twice? The skill answers with evidence. It compares row counts, null and duplicate keys, an additive total when one exists, and a SHA-256 hash of every material column in business-key order.
We tested the skill on PostgreSQL
We launched an isolated PostgreSQL 18 database on Railway and loaded three days of trip data.
The source contained 15 events for 13 trips. Two trips had late corrections. Trip 1003 changed from $30 to $12, and trip 1007 changed from $50 to $0 after a refund. The target had one row per trip but still held the old values.
The source keeps both versions. A naive append would keep 15 rows and count $429 in fares.
We then gave a fresh agent only the published skill, the table names, the three partitions, and the key and ordering fields. We did not give it the expected row count, total, hashes, or SQL.
The normal run
The agent inspected the database without changing it. It found the duplicate source keys, confirmed that the target had a primary key on (trip_date, trip_id), and checked that no trip moved between partitions.
It built two stage tables in separate transactions. Both had 13 rows, no null or duplicate keys, $349 in fares, and the same hash for every partition.
Then it showed the source fingerprint, transform, stage table, row counts, totals, and hashes. It did not treat our original request as permission to replace data.
After we approved that exact packet, the agent replaced one partition per transaction. Before each delete, it rechecked the approved stage. After each commit, it verified the target.
The target ended with 13 logical trips and $349 in fares. Running the backfill again from the same snapshot produces the same result.
The unsafe run
Next, we added a source view that put PostgreSQL's clock_timestamp() into a material target column. The row count stayed at 15, but the value changed on every read.
The agent read the view twice before staging and got two different source hashes. It refused to build a stage, produced no approval packet, and left both targets unchanged.
What this demonstrates
This was a 15-row synthetic test, not an Uber-scale workload. It still tested the behavior we care about.
The skill identifies late versions with a stable key and explicit ordering.
Two independent builds must match before approval.
Approval applies to exact fingerprints.
Replacement stays inside the named partitions.
A changing source stops the run before the agent touches the target.
One verification detail matters. PostgreSQL can store the same logical rows differently after a rewrite. The skill therefore compares ordered material values and their canonical hash, not raw bytes.
How Belvedere uses it
Belvedere uses agents to build pipelines, not act as the pipeline. When a building agent repairs published data, it can follow the same checks described here.
At enterprise scale, the catalog and data contract supply the keys, partitions, and ordering fields. The platform gives the agent scoped credentials, so there are no accidental writes.
Data owners see the approval packet before any partition changes. Its fingerprints, row counts, and totals become the audit record after replacement.
Teams can add checks from their own data contracts. A finance team might require the total to reconcile with its ledger. A team handling government data might restrict the work to one releasability boundary. The backfill workflow stays the same. The table owner decides which checks must pass.
Try the skill
curl -o SKILL.md https://www.clearfracture.ai/skills/idempotent-backfill.md
Keep DATABASE_URL in your environment or secret manager. Never paste the connection string into chat. Then give your coding agent the instruction below.
Apply the idempotent-backfill skill. Use the connection already set in
DATABASE_URLand never print it. RebuildTARGETfromSOURCE_SNAPSHOTforPARTITIONS. UsePARTITION_KEY,BUSINESS_KEY, andORDERING_FIELDS. Build twice, show me the fingerprints, and stop for approval before replacing the target.
Book a demo to see the same backfill checks inside Belvedere.
