Stay Ahead in Data Engineering

We build AI systems for a living. In production today, that means LangGraph wired into Belvedere: governed pipelines, human approvals, audit trails, provenance, etc.

But for this project, I wanted to kick the tires on something new: Flue, the agent framework from the Astro team. I’ve long been a fan of Astro for web development, so when they released Flue, I wanted to take it for a spin.

Initially I went down the path of adding Flue to background tasks (bug ticket sync, feedback triage, opportunity discovery, etc), but then decided to build something a little more fun, an 'agentic analyst org' powered by Flue.

It's completely free to use if you want to head to https://www.belvederelabs.ai/ and try it out with your data.

Flue Analyst Org | Belvedere Labs: drop a CSV and get an analyst's answer in a couple of minutes.

The data: 3,759 international matches assembled by Belvedere

We did not hand-roll the dataset. We built it in Belvedere, the same governed pipeline system we use in production.

The pipeline ("World Cup International Match Dataset") is eight nodes and seven edges in the canvas that was assembled by connecting our source API and using the following prompt:

"Pull senior men's international football fixtures and per-match statistics from the API-Football REST API, then computes leak-free pre-match features with all available statistics"

OpenTofu and Terraform solve nearly the same day-to-day problem. For Belvedere's current stack, AWS, EKS, Kubernetes, Helm, and Git-based CI/CD, the normal workflow is effectively equivalent: write HCL, use providers and modules, plan changes, apply through controlled automation.

The meaningful differences are licensing, governance, managed-service alignment, and how each tool handles state and plan artifacts.

For Belvedere's greenfield infrastructure work, OpenTofu fit the constraints we cared about most: open-source licensing, tool-layer state and plan encryption, active community governance, and portability across commercial, regulated, and customer-controlled environments.

Terraform remains a mature tool with the larger ecosystem, deeper HCP Terraform integration, and Terraform Stacks for teams centered on HashiCorp's managed control plane. Our decision was narrower: Belvedere was starting fresh, and OpenTofu gave us the Terraform-style workflow without taking on Terraform's current licensing and vendor-alignment tradeoffs.

Why This Matters

Infrastructure-as-code is not just deployment scripting. It becomes the control plane for cloud accounts, Kubernetes clusters, network boundaries, IAM, secrets-adjacent configuration, and recovery.

The Data Mesh promised a revolution: domain teams owning their own analytical data as high-quality, self-serve products. No more central bottlenecks. No more months-long waits for the right dataset. Just autonomous domains delivering trustworthy data that consumers could actually use.

Yet in practice, many organizations watch their mesh fracture—not because the technology fails, but because the surrounding organizational structure and politics quietly erode its utility. Processes emerge that look like “governance” on paper but function as guardrails limiting what data consumers can discover, interpret, or act upon. The result is a decentralized architecture that, in reality, recentralizes control in subtler ways.

This isn’t accidental. It’s the predictable outcome of misaligned incentives and communication structures. In this post we’ll examine why Data Mesh was created, how organizational theory explains its common fractures, the specific mechanisms that limit consumer power, and—most importantly—how agentic data engineering can realign roles so the mesh finally delivers on its promise.

Why Data Mesh Was Introduced in the First Place

By the late 2010s, enterprises were drowning in data lakes and warehouses. Central analytics teams had become bottlenecks: every new consumer request required pipeline changes, schema approvals, and months of engineering time. Data quality suffered. Delivery slowed. Domains closest to the source data had the deepest knowledge but no ownership or incentive to maintain analytical products.