Pangram 4.1: what the process does and does not do (2026-10-20)
Context
The process on the main page was developed against the model that identified itself as Pangram 4.0 on 2026-10-11 and 12. Responses now carry version: "4.1", readout_policy_id: pangram-4.1-continuous-readout-sentences-v1 and calibration_table_id: p42-120b-invariance-step24516-sentence-calibration-v2. The behaviour below is against 4.1.
The tester's critique, confirmed
A Pangram tester reported that on a full-length journalist reply the process stalled around 0.75 and never reached "Human", because an edit was only accepted if it lowered the highest segment score, and three segments sat at 1.000. That is correct. On a 341-word AI-generated source-request reply (landscaping business, cash-flow question, signed "Marcus"), 4.1 returns one 349-word segment at exactly 1.000. No single-sentence edit changes the displayed score, so the search has no gradient.
Three repairs tested
| repair | what it does | result on the 341-word email | cost |
|---|---|---|---|
| Continuous objective (the tester's suggestion, taken further) | optimise the word-weighted mean of −log10 P(human), read from class_probabilities[0], which is returned to ~1e-19 precision even when the score reads 1.000 |
one edit moved P(human) from 10^-29 to 10^-18.5; after that every candidate for every other sentence sat at 10^-18 ± 0.3. The document probability behaves like a maximum over sentences, so fixing one at a time cannot move it while several are tied | 156 units, ~$8 |
| Divide and conquer | optimise each paragraph standalone (where sentence labels are visible and each flip counts), then reassemble | 8 of 9 body paragraphs reached "Human" standalone in 4 to 36 units each. The reassembled email scored AI, fraction_human 5e-13, one 282-word AI segment. Paragraphs that pass alone fail together | 106 units, ~$5 |
| The original max-score search | per-sentence deletion ranking and rewrite | no gradient, as the tester found | stopped |
Re-check of earlier results under 4.1
| text | under 4.0 | under 4.1 |
|---|---|---|
| roof email, hand-edited (fold facts into parentheticals) | 0.286 Human | Mixed, fraction_ai 0.67 |
| roof email, search output (lean process) | 0.170 Human | Human, fraction_human 1.0 |
| restaurant email, search output | 0.345 Human | Human, 0.998 |
| restaurant email, search output continued | 0.152 Human | Human, 1.0 |
| roof email, original | 0.805 AI, one segment | AI, four segments scored 0 or 1 |
Conclusion
Under 4.1, the score-guided search still flips texts of roughly 130 words and under, and three earlier outputs still pass. It does not reach "Human" on a 300-plus-word email by any of the three methods above, and no further method is proposed. Context dominates: nine paragraphs that each read Human on their own read AI together, which means the document-level readout is doing work that a sentence-by-sentence search cannot see around.
What to patch
windows[].class_probabilitiesis returned to ~1e-19 precision. That is what gave the log-probability method its initial gradient. Round it, or drop it from the external API, as the displayed score already is.- The attack still works on short texts because a 130-word document has few sentences and the readout has little context to lean on. Any integration that scores emails or messages below ~150 words is exposed to the process on the main page, at $2 to $7 per text.
- Near-duplicate submission detection (shingle similarity above 0.9 between consecutive submissions from one key) would stop both the short-text attack and any future variant of the long-text one, independent of model changes.
Artefacts
evolve_v4.py (log-probability objective) and evolve_v5.py (per-paragraph) on the code page. The 341-word test email and every intermediate score are in the project cache.