I built a language with zero training data and an LLM wrote it at 85% first-try accuracy. Here's why.

Published July 2026 · PFCL on Codeberg

A few weeks ago I was using an LLM to write PFCL code — a real, working functional language, own type system, own runtime, own standard library. As far as any LLM's training data is concerned it doesn't exist.

I expected it to hallucinate function names and get the syntax subtly wrong, struggle the way every "new language vs LLM" post says it will. It didn't — performed almost flawlessly. So I went looking for why, and that turned into a much bigger thing than I planned: the full 164-task HumanEval benchmark, Claude and OpenAI both, PFCL against Python, across several different levels of context. This post is what I found.

The setup

PFCL doesn't have a fixed standard library. Every function — list.map, string.contains, whatever — lives in a catalog, identified by the hash of its own source, nothing baked into the compiler. If a function doesn't exist you add it to the catalog and it's there for good. (Longer version of why here if you want it — making the module system a hash map instead of the usual package/import setup.)

I built a harness that runs the same tasks through PFCL and Python, same models, same conditions. pass@1 is got it right first try, no help. pass@≤4 is got there within 4 tries, each retry seeing the real compiler error from the last attempt — actual error-driven repair, not blind re-guessing.

The result

Full context — grammar, a semantics doc, the whole catalog, all handed to the model up front — PFCL landed at 80-85% pass@1 and 95-100% pass@≤4, across both models. Python, same tasks, hit 87-88% and 100%. A language with zero presence in any training data came within a few points of a language that's in every training set there is.

That's the headline. The why took longer to work out.

A catalog you can fix beats documentation you can only hope works

Every LLM carries priors from every language it's seen, and point it at something new and it reaches for names from whatever looks closest — Haskell, OCaml, Python, C. Saw this constantly. PFCL's syntax is deliberately minimal and ML-style, lambdas and let and pattern matching, fast to pick up but that also means the functional-language instincts leak in whenever the catalog doesn't have what the model's reaching for.

With a fixed stdlib that's the end of it — write better docs, add warnings, hope. With a catalog there's a second move: check whether the wrong guess is pointing at a real gap, and if it is, add the function, and the model's wrong instinct is now correct, permanently, for every model after this one.

One task needed an MD5 hash. Both models failed it completely, every run, the whole project — not a wrong answer, just burning the entire retry budget and producing nothing. I checked the actual Rust source instead of the documented catalog and the function was already there: implemented, registered, working. It just never got a catalog entry, so no model could see it existed. Both models guessed at names — hash.md5, crypto.md5 — failed, then started hand-rolling a full MD5 implementation out of bitwise primitives instead of considering the function might exist somewhere they hadn't looked. That's what burned the budget.

I added the one missing file. Both models solved it on the next attempt, immediately.

Python can't do this. If a model's confused about something in hashlib that confusion is permanent, baked into millions of training examples, not something anyone gets to touch. A catalog you can just fix.

Models don't take the correction even when you hand it to them

Didn't expect this one. PFCL has a function that's a plain value — you reference it, you don't call it — one small exception to how basically every other language does this. I put an explicit warning about it directly in the context. The model called it as a function anyway, first try, ten out of ten runs.

Stranger one: a task needed a specific number of parameters, one model got the count wrong, and the error message stated the correct count explicitly, in plain text. Same mistake, all four retries, never adjusted, with the right answer sitting in the error every single time. The other model read the same error and fixed it on the very next try.

Same task, same error text, different model, different result. Some of what looks like "the language is hard for LLMs" is really just one model's blind spot, and you don't see that without testing more than one model on the same target.

What mattered in the context

Stripped pieces of the context and re-ran everything to check. Grammar spec gone — barely moved anything. Semantics doc gone — also barely moved. Catalog gone — dropped 40 points.

I'd assumed the documentation was doing the work. It's the opposite — a good per-function reference with real usage examples ends up teaching syntax as a side effect of just describing what the function does, better than prose explaining rules ever manages.

There's a placement effect worth its own paragraph. I had a warning about that value-vs-function mistake sitting in the general rules doc and it barely helped. Moved the same sentence into the specific catalog entry it concerns — right where the model looks when it's about to use the thing — and first-try success went from 0% to 20%, then to 40% once it became a before/after example instead of a plain warning. Same words. Just moved.

Where it doesn't work

Models mix up a wrapped value with a bare one — return Just(x) when the task wants x — on both models, on unrelated tasks, including the MD5 task above on the way to the eventually-right answer. Nothing missing from a catalog here. Just a reasoning slip that retries mostly paper over but don't fix.

Int/Float tracking is worse — the model loses track of which type a value should be partway through a calculation. I added guidance for exactly this and re-tested twice on the same task: fixed it once, failed completely the second time. Same guidance, same task.

And one task never got solved at all, either model, the whole project, no matter what I changed — burns its retry budget on generation length every time, produces no visible content. I don't have an explanation for that one.

None of this is a naming gap. It's what's underneath the naming gaps, and adding functions to a catalog doesn't touch any of it.

What happens once the catalog isn't small

All of the above is a catalog of 230 functions. That's small. I got curious what happens at 10,000.

Did the math and it's bad news for "just put the whole catalog in context." Extrapolating from the real numbers, 10,000 functions needs something like 3.8 million tokens just for the reference material, and Claude's 1-million-token window — already enormous by the standards of a couple years ago — runs out around 2,600 functions. Past that the approach stops, not gradually.

Stripping catalog entries down to only the fields that matter for writing a correct call buys some room: measured it, 70-85% smaller. Pushes the wall to maybe 10-15,000 functions, doesn't remove it.

The real fix is probably a "hot" subset loaded up front plus a way to search for anything else — the Hoogle move, or what a decent IDE's autocomplete already does. MCP is the obvious protocol for actually building that — resources for on-demand catalog search, a tool wrapping the real compiler for execution feedback — though I haven't built it yet. Sounds right architecturally, and it's also exactly the MD5 failure again, except guaranteed instead of occasional. At 230 functions with everything loaded that was one rare edge case. At scale most of the catalog is cold by construction, so "can't see the function I need" stops being the exception and becomes the normal state. Whether models reach for a search tool instead of inventing a plausible name again, I don't know yet.

The other thing I'm poking at

What if there's no documentation at all, just an editor connection. PFCL has a real VS Code extension backed by a language server — parse errors, completions, hover text, name-correction hints. No grammar file, no rules doc, no catalog dump, nothing but what the editor tells you as you type.

My guess is this splits. Pure syntax and "does this function exist" questions probably go fine — LSP diagnostics are a tight, well-shaped feedback loop, maybe tighter than most compilers give you. But anything that parses clean and resolves clean while still being semantically wrong — using a value the wrong way, an edge case the type checker doesn't catch yet — nothing would flag that, because there's no diagnostic for syntactically-fine-but-conceptually-wrong. Haven't run this yet. Needs a model with zero PFCL exposure connected to nothing but the editor, which is a real setup, not something to improvise mid-post.

Why this is worth writing down

PFCL itself is probably not the interesting part of this investigation. It's one language sitting in an odd spot — zero training exposure like any niche language, but built in a way that let me dig into why niche languages struggle instead of just confirming they do. The retry-loop finding in particular lines up almost exactly with an unrelated paper doing formally-verified synthesis with Dafny, proof verification instead of a compiler as the correction signal, same ordering: no context worst, static context helps, iterative feedback wins by the widest margin every time. Two unrelated setups landing on the same answer is worth more than either alone.

If you maintain a niche language and have wondered whether the LLM problem is really about training data, I'd like to compare notes.