Stop Trying to "Fix" Broken LLM JSON. It’s Poison.

Let’s have an honest conversation around the digital water cooler. How many hours of your life have you spent writing regular expressions to try and salvage a piece of JSON that an LLM mutilated?

You know the drill. You ask GPT-4 for a nice, clean Python dictionary of customer sentiment scores. It thinks for a while, spits out 500 tokens of absolute gold, and then… forgets the final closing curly brace. Or maybe it adds a trailing comma that chokes Python’s json.loads() like it’s swallowing a Lego.

Here at Factal, we used to view this as a parsing challenge. We built elaborate try/except blocks. We wrote regex that looked like ancient Sumerian curses just to strip out those pesky Markdown backticks. We thought we were being clever engineers, building "robust pipelines" that could handle the inherent messiness of stochastic parrots.

Well, we were wrong. We weren’t building robust pipelines; we were building sophisticated garbage chutes.

The "Canary Hypothesis"

We recently fell down a rabbit hole of machine learning research papers—specifically, a fascinating one about solving a million-step Tower of Hanoi problem without errors. (Yes, we read ML papers for fun. Don’t judge us.)

The researchers in that paper, and several others published recently, stumbled upon a hard truth that changed how we build data tools. They found a massive correlation between syntax errors and logical hallucinations.

The theory is known as the "Format-Reasoning Trade-off," but we like to call it the Canary Hypothesis.

Think of the LLM’s "brain" as having a limited amount of cognitive bandwidth. It has to juggle two things simultaneously:

1. The Reasoning: Solving your actual data problem (calculating a metric, writing SQL, summarizing text).

2. The Formatting: Remembering to close brackets, escape quotes, and adhere to strict JSON syntax.

When the task gets hard—when the reasoning demands 99% of that brainpower—the model starts cutting corners. And the first thing to go is the syntax.

The research suggests that a missing curly brace isn't just a typo. It’s a red flag waving frantically, signaling, "I got confused! My attention mechanisms failed! I have no idea what I'm doing!"

If the model was too overwhelmed to remember a closing bracket, do you really trust the complex SQL query inside that broken bracket?

Stop Polishing Turds

Before we read this research, when our parser choked, our instinct was: "Fix the parser."

Now, our instinct is: "Burn the output."

We realized that by building robust parsers to "heal" malformed output, we were actively masking hallucinations. We were successfully extracting incorrect data, putting a suit on it, and sneaking it into our dashboards. That is a data analyst’s nightmare.

If an LLM gives you broken output, it is trying to tell you that it failed the task. Listen to it.

The New Protocol

We’ve officially stopped trying to be clever with parsing. We are embracing stupidity. Our new internal rule for LLM pipelines is simple:

If json.loads() fails on the first try, discard the entire response.

Do not pass Go. Do not collect $200. Do not write a regex to fix the trailing comma. Throw it into the digital incinerator and sample again (perhaps with a higher temperature or a clearer prompt).

It feels counterintuitive to throw away "mostly good" answers. But trust us (and the research): if the wrapper is broken, the candy inside is poisoned. Save your regex skills for something else, and stop letting hallucinated data sneak into your pipelines disguised as a syntax error.