Practical Answers to Hard Data Challenges

When I became a CIA operations officer, I wasn’t thinking about data pipelines.

I was thinking about people, information, and the mission. About understanding what was happening and getting the right information to the people who needed it.

But somewhere behind every useful piece of intelligence was a less glamorous reality: someone had to get the information collected, processed, connected, and delivered.

Nobody puts that part in the movie.

They probably should. Although “We finally resolved the schema mismatch” would be a difficult line to build a trailer around.

The People Behind the Information

In the field, I relied on talented people to help get me good information. I was fortunate to have them.

They needed reliable ways to ingest and prepare data so analysts could connect the dots and mission teams could act. That meant more than getting information quickly. It meant preserving context, understanding where it came from, and having confidence in what it actually told us.

A useful connection buried in an inaccessible system was still buried. Information that arrived after a decision had been made couldn’t inform that decision.

The people behind that work made an enormous difference. Often, their effort was invisible precisely because they made things work.

TypeSafe released Jev this month, and it is a different kind of model. Where a language model writes you a paragraph, Jev takes a typed question and hands back a probability for each answer. We wanted to explore the technology, so we gave it a fun job: reading the news.

The result is Outliers, a small Belvedere Labs project that watches 88 news outlets for the stories they tell most differently. Jev reads each outlet's headline and teaser with the outlet's name removed and answers the same five questions, and the stories where the newsrooms disagree most rise to the top. The feed refreshes every few minutes.

What Jev Does With a Headline

Every headline and its teaser go to Jev with the same five questions.

• How alarmed is the wording, from reassuring to sensational?
• Does the headline state a fact, attribute a claim, ask a question, or offer an opinion?
• Who, if anyone, does it hold responsible?
• Does it promise more than its teaser delivers?
• Is the language loaded?

Because Jev answers with probabilities rather than prose, the rest is simple math. We add up how far apart the outlets sit on each question, give every story a divergence score, and post anything over 30. The grid at the top of this page shows the tone answers, one row per story and one column per outlet, and a brighter square is a more alarmed headline.

A Common Engineering Problem

Maybe you’ve been there: you find yourself given access to a giant, half-documented store of data, and you’re not entirely sure what you even have. Even worse, maybe you’ve never seen most of it before! Your unfortunate task is to find the needle in the haystack on a tight deadline. Connections are fragile, and you can’t risk altering any of the data, but you need answers fast. Where do you begin, and what can you realistically hope to achieve?

A tough situation to be sure, but one that is fairly common. Here at Clear Fracture we found ourselves running into this too. That’s why we developed Investigate, a new feature available in Belvedere. Investigate is a read-only research agent that answers questions about your data, grounded in your catalog.

How Investigate Can Help

To start digging into a task like this one, normally you at least need to have an idea of what databases and tables might be available, along with their schemas. Lineage is likely to play a role too, so that you can understand both upstream and downstream data flows around a particular asset.

With Investigate, you have the luxury of letting an agent figure those things out and document them for you. Instead of starting with SQL queries, you start with plain English: