BerylAI
Head-to-head · 58 scored questions · same prompts to both
BerylAI vs Claude
Both models got the same frozen set of questions. A program scored every answer against a known result, and a judge re-checked the general-knowledge answers the program marked wrong. Every question and both answers are on this page, including the ones BerylAI got wrong or declined.
- CPU only, no GPU
- Has read 33,114 files (about 550 MB of code and docs)
- Quotes the file and byte offset it read
- Maths answers carry an exact certificate
- Median reply 0.3 s
Scoreboard
RightWrongSaid it didn't know
Where BerylAI led
- Project memory, 4 of 6 vs 0 of 6. Questions about the project's own lab records. BerylAI has read them and quotes the file and byte. Claude has never seen them.
- Exact strings, 12 of 12 vs 11 of 12. Reverse, double or sort strings up to 300 characters, checked character by character.
- Maths, 10 of 10, tied. BerylAI's proofs come with a certificate checked in exact arithmetic, and its "false" answers name a counterexample.
Where Claude led
- General knowledge, 30 of 30 vs 6 of 30. BerylAI only answers from what it has read, so it declined 23 of these. It got 1 wrong.
- Open writing. Claude wrote a full answer to all five. BerylAI declined four, and its one attempt (a 5M-parameter text model) is not readable.
1Claude answers stated and wrong
3BerylAI answers stated and wrong
Code memory, run live
Four prompts typed into BerylAI on 21 November 2026, each checked against the real source file. Claude was not run on these four.
Every question, both answers
How it was scored
- The question set was frozen before either model answered: 30 general-knowledge questions drawn at random from the NQ-open validation set, 12 exact string tasks, 10 inequality claims, 6 questions about the project's records and 5 open writing prompts.
- Claude's answers were collected on 10 October 2026. BerylAI's answers are from its run of 27 October 2026. Nothing was edited afterwards.
- A program scored everything except the open prompts. General knowledge is right when the dataset's answer appears in the reply. Strings must match exactly. Maths must say TRUE or FALSE correctly. Project questions must contain the recorded number.
- The NQ dataset's answers date from 2018, so some correct answers look wrong to a program. A judge re-checked every general-knowledge answer the program marked wrong. These show as accepted with the judge's reason. All ten of Claude's held up. BerylAI's one did not.
- "Said it didn't know" counts as neither right nor wrong. BerylAI declines whenever nothing it has read answers the question.
- The open writing prompts are not scored by a program. Read both answers and judge for yourself.
- BerylAI is the public name for the model called Engram in the project's logs.