Method & benchmarks
Numbers that know where they stop.
By ShakuPublished Updated
How Kaneme measures a voice, how we test the engine, and where it refuses to answer.
Published results
You, or generic AI writing
0.989
AUC against generic AI (Q1)
You, or another human
0.869
AUC against another human (Q2)
- Validated language
- French, English
- Served engine
- Nemi EN 1.1 (calibration emb-en-5.0); Nemi FR 1.1 (model voice-fr-base, calibration emb-frq-1.1)
AUC: 1 = perfect separation, 0.5 = chance. Each protocol is in chapter 2.
Method contents
- 01What we measureTwo questions, ranked by importance.
- 02ResultsInternal benchmark, public benchmarks, what a score is worth.
- 03How we testFrom corpus to publication, in five steps.
- 04Where it breaksThe three limits that change an answer.
- 05Product benchmarkThe public comparison, to play yourself.
- 06Experimental detailsCalibration, adversaries, all limits.
01
Two questions. Never one blended score.
“Is this you rather than another person?” and “is this you rather than a machine?” need different evidence. At a score of 80, the calibrated probability against AI is 99%; the comparison against another person returns an indicative reading. Kaneme keeps them apart.
Your voice, or generic AI writing?
The English benchmark compares real authors with generic AI writing. Deliberate imitation is a separate challenge and is not covered by this result.
You, or another human?
Much harder: people in the same field writing about the same subjects naturally overlap. Kaneme publishes this weaker result because it describes the difficult cases.
Writing-pattern review. It describes a draft’s rhythm, vocabulary, phrasing and avoided words, without attributing authorship.
Free test (Q0). Without your reference it answers a weaker question. Its scores and weaknesses are in the experimental details.
Definition
Voice measurement
Voice measurement evaluates how closely a text matches a reference voice built from its author’s writing. Without a reference corpus, Kaneme does not display it.
A high measurement says that the text resembles your reference in the dimensions Kaneme covers. It does not identify who used the keyboard, which model assisted, or whether the result can establish authorship.
What it measures
- similarity in rhythm, syntax and punctuation to your reference
- consistency in vocabulary, signature words and phrases you tend to avoid
- the distance between the evaluated text and your real writing
- overall fidelity within the genres and lengths supported by the engine
What it does not measure
- your voice without a reference corpus
- the certain origin of a text or the tool that produced it
- a universal “human versus AI” probability
- legal proof of authorship or a promise of undetectability
Questions about voice measurement
What exactly does voice measurement cover?+
Voice measurement evaluates how closely a text matches a reference voice built from its author’s writing. Without a reference corpus, Kaneme does not display it.
Why do I need to provide my own writing?+
Fidelity only exists in relation to a reference. Kaneme builds that reference from texts you confirm you wrote, then compares the evaluated text with their recurring style patterns.
Is this an AI detector?+
No. An AI detector tries to infer a text’s origin without knowing its author. Voice measurement asks a narrower question: does this text resemble the saved reference voice? It proves neither human authorship nor the absence of AI assistance.
Can a voice be measured without my own writing?+
No. Without a reference corpus there is nothing to compare: reading one text alone can name its generic patterns, but that is not voice measurement. Voice measurement requires your reference voice.
Does a high measurement establish that I wrote the text?+
No. It indicates proximity to your reference within the measurement’s supported domain. The stamp adds dedicated contrasts and may abstain; neither result is legal proof of authorship.
02
The result has to survive unseen authors.
We hold authors back before tuning, test on their real writing, then rerun the bench for every engine version. If a number falls, the published number falls with it.
Internal benchmark
0.989
Your voice versus generic AI writing
The English evaluation compares held-out authors with generic AI writing. It does not establish resistance to deliberate imitation.
0.869
Your voice versus another human
The harder question, published because it describes the difficult cases. Measured on a sealed exam with people never seen before.
99%
What a score of 80 is worth
The same displayed score does not carry the same evidence against AI and against another human.
Public benchmarks · PAN (CLEF) and AuthBench
The unchanged engine is also evaluated on public authorship competitions: texts we neither wrote nor chose. These out-of-domain results are a floor, not the product score.
These benchmarks were run by the previous engine version. The current version has not rerun them yet, so no number is shown rather than copying one from an engine that is no longer served.
03
How we test
Five steps, always in this order. A result that skips one is not published.
1Corpus
English writing from separate authors for training, calibration and the final held-out evaluation.
2Held-out set
Fresh authors never seen in training, read once, at the end.
3Calibration
Every score is checked against reality, pair by pair, separately against AI and against humans.
4Test
AUC, false positives and attacks, measured out of sample, genre by genre.
5Publication
Published numbers follow the served engine in the same deployment, including when they drop.
04
Where it breaks
Kaneme is not a universal AI detector: it measures whether a text sounds like a reference voice. This is where that measurement stops.
The reasoning behind these boundaries is set out in the manifesto.
Outside its scope, it abstains
Too little text, a thin corpus or a register the engine cannot read produces no forced score and no charge.
Writing genre matters
Results are published genre by genre, because some genres overlap far more than the median suggests. The comparison against another human returns an indicative reading in French; in English it names the author only above the threshold of its sealed exam.
So does language
English and French each have their own engine: trained, calibrated and validated on authors writing in that language. The question “you or another human?” is measured in both.
05
The best benchmark is your own voice.
The numbers on this page describe corpora you do not know. To judge for yourself, give three of your texts, then another one to measure: the stamp says whether it carries your voice, or abstains when it cannot decide.
Going furtherSee the experimental details: calibration, adversaries and all limits
A score still isn’t a probability
Kaneme checks what each score meant on held-out writing, separately against other people and against generic AI writing. Results from French adversarial tests do not establish English resistance to evasion.
The detailed evidence stays with its valid domain
French genre tables, free-test weaknesses and provider rates remain on the French page. They do not describe the English engine.
Not a universal AI detector
Kaneme measures similarity to your reference. It does not infer the absolute origin of any text.
Adversarial performance is separate
A strong result on ordinary samples does not prove resistance to deliberate evasion. Kaneme publishes that distinction.
Short texts are asymmetric
Measured on the served engine: at 70 words, none of the 600 generic AI texts in our test reached the top band: a very high score keeps its meaning on a short text. The measure is also weaker there: it ranks the author’s text above a generic AI text 93 times out of 100 at 70 words, versus 99 at about 190 words. A weak signal on a short text, on the other hand, cannot prove a mismatch: it is an absence of evidence, not evidence of absence.
Verdicts and certificates use different displays
The public verdict can show a calibrated percentage with its question, confidence and limits. Public certificates show a band and provenance rather than a fine score. Neither display alone proves who wrote a text.
Bring the writing. Let the evidence answer.
Start with 100 words in the passport, enrich your voice with references and perspective, then compare each new text in the same language.