BerylAI crystal mark
BerylAI
Head-to-head · 58 scored questions · same prompts to both

BerylAI vs Claude

Both models got the same frozen set of questions. A program scored every answer against a known result, and a judge re-checked the general-knowledge answers the program marked wrong. Every question and both answers are on this page, including the ones BerylAI got wrong or declined.

Scoreboard

RightWrongSaid it didn't know
Where BerylAI led
  • Project memory, 4 of 6 vs 0 of 6. Questions about the project's own lab records. BerylAI has read them and quotes the file and byte. Claude has never seen them.
  • Exact strings, 12 of 12 vs 11 of 12. Reverse, double or sort strings up to 300 characters, checked character by character.
  • Maths, 10 of 10, tied. BerylAI's proofs come with a certificate checked in exact arithmetic, and its "false" answers name a counterexample.
Where Claude led
  • General knowledge, 30 of 30 vs 6 of 30. BerylAI only answers from what it has read, so it declined 23 of these. It got 1 wrong.
  • Open writing. Claude wrote a full answer to all five. BerylAI declined four, and its one attempt (a 5M-parameter text model) is not readable.
1Claude answers stated and wrong
3BerylAI answers stated and wrong

Code memory, run live

Four prompts typed into BerylAI on 21 November 2026, each checked against the real source file. Claude was not run on these four.

Every question, both answers

How it was scored