--- title: "Articles - Data Engineering Insights & Best Practices" description: "Practical advice for data teams on reducing pipeline bottlenecks, automating manual work, governing data products, and delivering insights faster." canonical: "https://www.clearfracture.ai/articles" --- From the Clear Fracture Team # Data Engineering in the Age of Agentic AI The landscape of enterprise data is shifting fast — AI agents are rewriting the rules of pipeline engineering, governance, and trust. We publish what we learn building at the frontier so your team can move faster and decide with confidence. Get the next issue Weekly. Written by practitioners. Unsubscribe anytime. [![Agent Skill: Idempotent Backfill for Late-Arriving Data](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1787677997000-idempotent-backfill-animation_2x.gif&w=1920&q=75)](/articles/an-idempotent-backfill-skill-for-agentic-data-pipelines) [ ## Agent Skill: Idempotent Backfill for Late-Arriving Data ](/articles/an-idempotent-backfill-skill-for-agentic-data-pipelines) ![Haydn Strauss](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1776107930464-E8BC84B9-E3E5-4B8B-B138-F2D6989C734A.png&w=48&q=75) [Haydn Strauss](/articles/author/haydn-strauss)4 min readData EngineeringPublished August 25, 2026 Uber built Apache Hudi to solve a familiar data problem: **Records change after they land.** - A trip gets corrected after it ends. - A chargeback arrives weeks later. Rebuilding a large table to fix a handful of rows is wasteful. Our free [idempotent backfill skill](https://www.clearfracture.ai/skills/idempotent-backfill.md) applies some of Hudi's design ideas to late-arriving corrections in data you've already published. Give the current URL to your coding agent and simply tell it to *install the skill* to try it out. ## What we borrowed from Hudi Uber's [lakehouse write-up](https://www.uber.com/us/en/blog/ubers-lakehouse-architecture/) describes a backfill that reads a fixed snapshot and overwrites only the affected partitions. It does not move the incremental writer's checkpoint. That separation prevents an old repair from disrupting the live pipeline. Our skill turns a few of its core practices into a six-step, database-independent checklist: 1. Fix the source snapshot. 2. Name the partitions, business key, and newest-wins order. 3. Build each partition twice. 4. Compare the logical rows. 5. Stop for approval. 6. Replace and verify one partition at a time. ![The six gates of the idempotent backfill skill](https://rgzgbulrzrbknuiprxti.supabase.co/storage/v1/object/public/media/uploads/1787679315617-published-skill.png?v=3) *The skill is short on purpose so the agent can use the database's own atomic replacement operation.* ## Why build it twice Maxime Beauchemin's essay on [functional data engineering](https://maximebeauchemin.medium.com/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a) gives us a clear rule. Treat a partition as the complete output of a function. The same source snapshot and transform should return the same rows every time. [Read whole article](/articles/an-idempotent-backfill-skill-for-agentic-data-pipelines) [![From Here to There: How Belvedere™ Maps Your Current State and Builds the Path to Your Target](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1786592161000-here-to-there-map-animation-v2_2x.gif&w=1920&q=75)](/articles/from-here-to-there) [ ## From Here to There: How Belvedere™ Maps Your Current State and Builds the Path to Your Target ](/articles/from-here-to-there) ![Brian Frutchey](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1775054766834-1613771332003.jpg&w=48&q=75) [Brian Frutchey](/articles/author/brian-frutchey)6 min readSystem ModelingPublished August 18, 2026 Every capable agent (human or artificial) needs two things before it can act with confidence: a clear picture of **where we are**, and a robust definition of **where we need to go**. That sounds obvious. It is also where most agentic systems quietly fail. We have poured enormous energy into making models smarter, tools more composable, and orchestration layers more sophisticated. Yet the hard problem is not reasoning in the abstract. It is grounding that reasoning in a faithful account of the present (*here*) and an unambiguous specification of the intended future (*there*). Without both, an agent is improvising. With both, it can plan, execute, verify, and explain. ## Agents Don't Need Magic. They Need Context with Edges. An AI agent assisting a mission, a business process, or a data pipeline is only as good as the situation it can see and the outcome it is asked to produce. "Current state" is not a chat transcript. "Goal" is not a vague aspiration. Both must be detailed enough that another competent actor (software or human) could inspect them, challenge them, and act on them. That means capturing: - **Here**: what exists now (systems, sources, constraints, policies, dependencies, quality, ownership, and known gaps). Not a slide. Not a tribal memory. Ground truth. - **There**: what "done" looks like (required outcomes, acceptance criteria, interfaces, governance rules, and the boundaries the agent must not cross). [Read whole article](/articles/from-here-to-there) [![Turn GitHub Repositories Into Explorable SysML v2 System Models](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1784667213851-automated-system-modeling-animation_2x.gif&w=1920&q=75)](/articles/sysml-repo-modeler-github-repositories) [ ## Turn GitHub Repositories Into Explorable SysML v2 System Models ](/articles/sysml-repo-modeler-github-repositories) [Jeremy Fields](/articles/author/jeremy-fields) and [John Sutton](/articles/author/john-sutton)6 min readSystem ModelingPublished August 5, 2026 SysML Repo Modeler is now open source and free to use. It turns one GitHub repository—or many—into an explorable SysML v2 system model. [Get the code on GitHub](https://github.com/ClearFracture/sysml-repo-modeler). It gives us a coherent way to visualize our own repositories, services, APIs, and layers of institutional knowledge in an exploratory systems view. And now it’s open source for anyone to freely use. ## Explore a Live Model Want to see the result before getting into the details? Start with the [live Supabase Platform model](https://www.belvederelabs.ai/project-analyzer/supabase-platform), which maps five repositories and four languages as one system. You can also explore the [OpenClaw model](https://www.belvederelabs.ai/project-analyzer/openclaw), the [n8n model](https://www.belvederelabs.ai/project-analyzer/n8n) and the [Ollama model](https://www.belvederelabs.ai/project-analyzer/ollama). All four sit side by side in the [SysML Repo Modeler gallery](https://www.belvederelabs.ai/project-analyzer), with the repository, part and connection counts for each. [![The SysML Repo Modeler model of Supabase Platform showing services and dependencies across five repositories](https://www.belvederelabs.ai/screenshots/supabase-platform.webp)](https://www.belvederelabs.ai/project-analyzer/supabase-platform) [*Open the live Supabase Platform model*](https://www.belvederelabs.ai/project-analyzer/supabase-platform) *— click the image to search, filter, and explore the system.* ## The Problem: Architecture Lives in Too Many Places The reality is that modern systems are scattered across multiple repositories, services, APIs, and layers of institutional knowledge. The challenge is not that teams lack documentation; it is that documentation struggles to keep up with what the code does. Traditional Model-Based Systems Engineering (MBSE) documentation can be useful, but it is often manually maintained across disparate software platforms that require significant user knowledge and training, with limited ability to transfer data between tools. As a system evolves, the diagram becomes a snapshot of what people thought a system looked like but not necessarily what exists now. [Read whole article](/articles/sysml-repo-modeler-github-repositories) [![Clear Fracture’s Belvedere™ Assessed “Awardable” for Department of War work in the CDAO’s Tradewinds Solutions Marketplace](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1785169572052-Belvedere-TSM-Awardable-Article-16x9.png&w=1920&q=75)](/articles/clear-fractures-belvedere-assessed-awardable-for-department-of-war-work-in-the-cdaos-tradewinds-solutions-marketplace) [ ## Clear Fracture’s Belvedere™ Assessed “Awardable” for Department of War work in the CDAO’s Tradewinds Solutions Marketplace ](/articles/clear-fractures-belvedere-assessed-awardable-for-department-of-war-work-in-the-cdaos-tradewinds-solutions-marketplace) 2 min readPress ReleasePublished July 27, 2026 **FOR IMMEDIATE RELEASE** **Vienna, VA — July 27, 2026 —** Clear Fracture LLC, developer of Belvedere™, the Agentic Data Manager, today announced that it has achieved “Awardable” status through the Chief Digital and Artificial Intelligence Office’s (CDAO) Tradewinds Solutions Marketplace. The Tradewinds Solutions Marketplace is the premier offering of Tradewinds, the Department of War’s (DoW’s) suite of tools and services designed to accelerate the procurement and adoption of Artificial Intelligence (AI)/Machine Learning (ML), data, and analytics capabilities. Belvedere puts AI agents to work as data engineers. Analysts and mission owners describe what they need in plain language; Belvedere’s agents discover the source data, design the transformations, and compile them into governed, production-ready pipelines that run on the organization’s existing infrastructure. The agents build the pipeline; they are not the pipeline. Every pipeline they produce is transparent, auditable, and repeatable, and it runs as ordinary code, keeping operations cost-efficient at mission scale. “Mission teams lose too much time wiring data together by hand, and the systems that result are hard to trust and hard to maintain,” said Brian Frutchey, Chief Technology Officer of Clear Fracture. “Belvedere’s agents do that engineering work in the open. Every pipeline they build can be inspected, audited, and run again tomorrow. Awardable status through Tradewinds gives DoW customers a direct path to put that capability on contract.” [Read whole article](/articles/clear-fractures-belvedere-assessed-awardable-for-department-of-war-work-in-the-cdaos-tradewinds-solutions-marketplace) [![A Write-Audit-Publish (WAP) Skill for Agentic Data Pipelines](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1784051508486-write-audit-publish-animation_2x.gif&w=1920&q=75)](/articles/a-write-audit-publish-wap-skill-for-agentic-data-pipelines) [ ## A Write-Audit-Publish (WAP) Skill for Agentic Data Pipelines ](/articles/a-write-audit-publish-wap-skill-for-agentic-data-pipelines) ![Haydn Strauss](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1776107930464-E8BC84B9-E3E5-4B8B-B138-F2D6989C734A.png&w=48&q=75) [Haydn Strauss](/articles/author/haydn-strauss)4 min readData EngineeringPublished July 14, 2026 AI agents are great at building data pipelines that look like they work until you dig into the results. Write-audit-publish (WAP) helps fix that. Stage the data, audit it against a declared contract, and only publish once every clause passes. Netflix popularized this pattern in 2017. A pipeline that finishes successfully is not the same as one whose output is correct. We’ve built a number of internal skills to make our own data pipelines safer, and this one felt useful enough to release as [a free WAP skill for coding agents](https://www.clearfracture.ai/skills/write-audit-publish.md). The first test was on Netflix’s [Top 10 dataset](https://www.netflix.com/tudum/top10). The initial run stopped at the gate. Our contract said every film should have `“N/A”` as the season title, but the agent found nine rows that didn’t match. The contract was wrong, not the data. We fixed it, started a fresh run, and the second attempt published cleanly, with the total reconciling to exactly 185,656,120,000 hours viewed. We ran it again on an NFL play-by-play pipeline (converting play description strings into structured stat tables). It caught a parser bug that left 1,723 completed passes without matching receptions, exactly the kind of thing a "successful" run hides. Below, we dig a bit more into how the skill works. Give it a read, or point your coding agent at this URL and try it yourself. [Read whole article](/articles/a-write-audit-publish-wap-skill-for-agentic-data-pipelines) [![The Foundation Behind Reliable AI Agent Analytics](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1781555473124-governed-layer-illustration.gif&w=1920&q=75)](/articles/the-foundation-behind-reliable-ai-agent-analytics) [ ## The Foundation Behind Reliable AI Agent Analytics ](/articles/the-foundation-behind-reliable-ai-agent-analytics) ![Haydn Strauss](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1776107930464-E8BC84B9-E3E5-4B8B-B138-F2D6989C734A.png&w=48&q=75) [Haydn Strauss](/articles/author/haydn-strauss)4 min readAnalyticsPublished June 16, 2026 Anthropic recently published [how it runs self-service analytics on Claude](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude). One result caught my eye: **context + skills took its analytics agent from 21% accuracy to consistently above 95%.** **Highlighting that generating SQL is the easy part,** the hard part is everything underneath it: canonical datasets, a semantic layer, lineage, maintained skills, and provenance on every answer. That jump came from the foundation, not a bigger model. With the context right, the agent on top matters much less. ## Why Agents Alone Fail In addition to cost, three context problems keep coming up. - **Entity ambiguity.** "Active users" or "revenue" has several definitions in the warehouse. The agent picks one and writes correct SQL against the wrong data. - **Staleness.** The definition was right when written. Then the pipeline changed and the skill was never updated. - **Retrieval failure.** The right definition exists somewhere, but the agent can't find it, or grabs the wrong version. Two of these, staleness and retrieval, can't be fixed easily by prompting alone. They need the context to be a versioned, owned asset wired to the pipeline it describes. Anthropic tried the shortcut of handing the agent the raw query corpus, and accuracy barely moved. As they put it: *"The information was there, the agent saw it, and it still didn't use it."* [Read whole article](/articles/the-foundation-behind-reliable-ai-agent-analytics) [![From Belvedere Pipeline to Flue Agents: A Skeptical Pick of the 2026 World Cup Winner](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1780934378582-flue-framework.gif&w=1920&q=75)](/articles/from-belvedere-pipeline-to-flue-agents-a-skeptical-pick-of-the-2026-world-cup-winner) [ ## From Belvedere Pipeline to Flue Agents: A Skeptical Pick of the 2026 World Cup Winner ](/articles/from-belvedere-pipeline-to-flue-agents-a-skeptical-pick-of-the-2026-world-cup-winner) ![Haydn Strauss](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1776107930464-E8BC84B9-E3E5-4B8B-B138-F2D6989C734A.png&w=48&q=75) [Haydn Strauss](/articles/author/haydn-strauss)9 min readAnalysisPublished June 9, 2026 We build AI systems for a living. In production today, that means LangGraph wired into Belvedere: governed pipelines, human approvals, audit trails, provenance, etc. But for this project, I wanted to kick the tires on something new: [Flue](https://flueframework.com/), the agent framework from the [Astro](https://astro.build/) team. I’ve long been a fan of Astro for web development, so when they released Flue, I wanted to take it for a spin. Initially I went down the path of adding Flue to background tasks (bug ticket sync, feedback triage, opportunity discovery, etc), but then decided to build something a little more fun, an 'agentic analyst org' powered by Flue. It's completely free to use if you want to head to [https://www.belvederelabs.ai/](https://www.belvederelabs.ai/) and try it out with your data. ! *Flue Analyst Org | Belvedere Labs: drop a CSV and get an analyst's answer in a couple of minutes.* ## The data: 3,759 international matches assembled by Belvedere We did not hand-roll the dataset. We built it in Belvedere, the same governed pipeline system we use in production. The pipeline ("World Cup International Match Dataset") is eight nodes and seven edges in the canvas that was assembled by connecting our source API and using the following prompt: *"Pull senior men's international football fixtures and per-match statistics from the API-Football REST API, then computes leak-free pre-match features with all available statistics"* [Read whole article](/articles/from-belvedere-pipeline-to-flue-agents-a-skeptical-pick-of-the-2026-world-cup-winner) [![OpenTofu vs. Terraform: Choosing an IaC Control Plane for Belvedere](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1780345518739-terraform-opentofu.gif&w=1920&q=75)](/articles/iac-tooling-opentofu-vs-terraform) [ ## OpenTofu vs. Terraform: Choosing an IaC Control Plane for Belvedere ](/articles/iac-tooling-opentofu-vs-terraform) ![Zech Crannigan](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1780343643662-1592665325450.jpg&w=48&q=75) [Zech Crannigan](/articles/author/zech-crannigan)5 min readProductPublished June 2, 2026 OpenTofu and Terraform solve nearly the same day-to-day problem. For Belvedere's current stack, AWS, EKS, Kubernetes, Helm, and Git-based CI/CD, the normal workflow is effectively equivalent: write HCL, use providers and modules, plan changes, apply through controlled automation. The meaningful differences are licensing, governance, managed-service alignment, and how each tool handles state and plan artifacts. For Belvedere's greenfield infrastructure work, OpenTofu fit the constraints we cared about most: open-source licensing, tool-layer state and plan encryption, active community governance, and portability across commercial, regulated, and customer-controlled environments. Terraform remains a mature tool with the larger ecosystem, deeper HCP Terraform integration, and Terraform Stacks for teams centered on HashiCorp's managed control plane. Our decision was narrower: Belvedere was starting fresh, and OpenTofu gave us the Terraform-style workflow without taking on Terraform's current licensing and vendor-alignment tradeoffs. ## Why This Matters Infrastructure-as-code is not just deployment scripting. It becomes the control plane for cloud accounts, Kubernetes clusters, network boundaries, IAM, secrets-adjacent configuration, and recovery. [Read whole article](/articles/iac-tooling-opentofu-vs-terraform) [![How Political and Organizational Friction Corrupts the Data Mesh](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1776716795467-data-mesh-fractured.gif&w=1920&q=75)](/articles/how-political-and-organizational-friction-corrupts-the-data-mesh) [ ## How Political and Organizational Friction Corrupts the Data Mesh ](/articles/how-political-and-organizational-friction-corrupts-the-data-mesh) ![Keith Schumacher](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1774967168751-belvedere-on-blue.png&w=48&q=75) [Keith Schumacher](/articles/author/keith-schumacher)7 min readBest PracticesPublished May 20, 2026 The Data Mesh promised a revolution: domain teams owning their own analytical data as high-quality, self-serve products. No more central bottlenecks. No more months-long waits for the right dataset. Just autonomous domains delivering trustworthy data that consumers could actually use. Yet in practice, many organizations watch their mesh fracture—not because the technology fails, but because the surrounding organizational structure and politics quietly erode its utility. Processes emerge that look like “governance” on paper but function as guardrails limiting what data consumers can discover, interpret, or act upon. The result is a decentralized architecture that, in reality, recentralizes control in subtler ways. This isn’t accidental. It’s the predictable outcome of misaligned incentives and communication structures. In this post we’ll examine why Data Mesh was created, how organizational theory explains its common fractures, the specific mechanisms that limit consumer power, and—most importantly—how agentic data engineering can realign roles so the mesh finally delivers on its promise. ### **Why Data Mesh Was Introduced in the First Place** By the late 2010s, enterprises were drowning in data lakes and warehouses. Central analytics teams had become bottlenecks: every new consumer request required pipeline changes, schema approvals, and months of engineering time. Data quality suffered. Delivery slowed. Domains closest to the source data had the deepest knowledge but no ownership or incentive to maintain analytical products. [Read whole article](/articles/how-political-and-organizational-friction-corrupts-the-data-mesh) [![Introducing Belvedere: A Control Plane for Reliable Data in the Age of AI](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1774967168751-belvedere-on-blue.png&w=1920&q=75)](/articles/belvedere-a-control-plane-for-reliable-data-in-the-age-of-ai) [ ## Introducing Belvedere: A Control Plane for Reliable Data in the Age of AI ](/articles/belvedere-a-control-plane-for-reliable-data-in-the-age-of-ai) ![Brian Frutchey](/_next/image?url=https%3A%2F%2Frgzgbulrzrbknuiprxti.supabase.co%2Fstorage%2Fv1%2Fobject%2Fpublic%2Fmedia%2Fuploads%2F1775054766834-1613771332003.jpg&w=48&q=75) [Brian Frutchey](/articles/author/brian-frutchey)5 min readData EngineeringPublished March 31, 2026 Everyone understands the power of data and its never-ending growth. Yet the amount of time we have to get the data we need stays the same. The "attention economy" is the result… and the most easily available data biases our decisions. The harder it is to gather and interpret data ourselves the more we allow third parties to control what we see. It is no secret that this leaves us open to manipulation. Consumers need to be put back in charge of their own destiny! But general data processing interfaces haven't changed much in decades (all hail SQL) and hoping a question-answering AI won’t hallucinate or be biased itself is asking for trouble – even if you can afford the tokens to process all the relevant data. So how can we reduce the months of work historically required to generate a new, reliable data product to minutes? Clear Fracture was founded to remove the barriers between users and the right data. We believe AI agents are a huge part of the answer, allowing attention to be scaled through compute. Not enough hours in a day to research deeply? Task an agent to do that work on your behalf. Can't extract a needed insight fast enough from a mountain of data? Direct a swarm of agents to chew through the data at cloud scale to deliver the insight in moments. Worried your opinion or decision is biased? Have agents argue amongst themselves until they reach consensus from all the perspectives represented in available data. When you are in charge of your own AI army, it becomes your armor against manipulation and short-sightedness. [Read whole article](/articles/belvedere-a-control-plane-for-reliable-data-in-the-age-of-ai) --- Source: [https://www.clearfracture.ai/articles](https://www.clearfracture.ai/articles)