This fall I had the chance to do something I had only read about: test a commercial AI safety system from the outside and publish what I found, good and bad. My previous research and article work generating synthetic scam conversations for COVA (Cognitive Operations Virtual Assistant) and COVA-X made it possible.

🤝 The opportunity

Tuteliq offers a content moderation API built around the harm categories of the U.S. Kids Online Safety Act, for platforms where minors interact online. Gabriel Sabadin of Tuteliq gave me the chance to evaluate the system independently. That meant I designed the tests, ran them and drew the conclusions, and Tuteliq reviewed the report only for factual accuracy. The work was unpaid, which kept it independent. Dr. Ayan Roy of Christopher Newport University joined as second evaluator.

🧪 What I built

Because no real person’s messages could be used, every test conversation is synthetic. Using an open-weight language model (Gemma 3 12B) running on my own workstations, I generated 1,294 multi-turn text conversations and 50 single messages across five harm categories: romance scams, social engineering, coercive control, bullying, and self-harm and emotional distress. Each category had five parts:

• harmful conversations written plainly;
• the same harm hidden or softened (evasion probes);
• the same harm disguised as something legitimate (adversarial probes);
• ordinary harmless conversations;
• harmless conversations built to look suspicious (hard negatives), to measure false alarms where they matter most.

Every pattern was grounded in published research or official guidance. Before testing, each set passed balance, shortcut and diversity checks, and I read samples by hand. Then I froze every set and fingerprinted every file, so any later change would be detectable.

⚙️ Running and scoring

The run went through Tuteliq’s own evaluation harness: 1,344 cases, each sent once, with no failed calls. I scored the saved responses offline and reported every rate with a 95% Wilson score interval, because the sets are small. Dr. Roy labeled a blind sample of 50 conversations and agreed with my labels on 41 (82%; Cohen’s kappa 0.64).

📊 What we found

Detection rate and false-alarm rate for each category:

• ✅ Romance scams: 100% detected (130/130), 6.9% false alarms
• ✅ Coercive control: 99.2% detected, 0.8% false alarms
• ✅ Self-harm and emotional distress: 96.1% detected, 0.8% false alarms
• ⚠️ Bullying: 84.5% detected, 4.6% false alarms
• ⚠️ Social engineering: 83.7% detected, 13.2% false alarms

Detection held up on the synthetic evasion and adversarial probes. The clearest gaps were in social engineering, the weakest category, where scams asking for identity data under a government pretext were mostly missed (7 of 28). Bullying that builds gradually from teasing was also detected less often (65.2%). Tuteliq approved publishing both the full report and the public summary, weaknesses included.

🔁 After the evaluation, Tuteliq was given royalty-free permission to use the test dataset for internal testing and improving detection, so their ML team can work directly on the gaps we found. Any future independent retest should use new, unseen conversations.

💡 What I learned

1. The hardest part was making harmful and harmless conversations differ only in the harm. On the harmful side, the model kept falling back on stock persuasion phrases, and some scam messages repeated wording from my own instructions word for word, which a detector could match instead of the scam itself. On the harmless side, ordinary money talk, like splitting a cost or paying someone back, had to be deliberately designed in; in social engineering, a few harmless setups pulled the model toward money talk on every try and had to be dropped. Before testing, I checked both sides for phrases and topics that appear on only one of them, and the report discloses the ones that remained.

2. “Harmless” is genuinely hard to judge, even for experts. Dr. Roy agreed with my labels on 41 of 50 conversations. Eight of the nine disagreements were harmless conversations he read as harmful, and in social engineering he judged four of the five harmless conversations to be scams. Ordinary messages from banks and services really do look like scams. That’s not a flaw in his reading; it’s the problem every detector faces.

3. False alarms tell you as much as detections. Most romance false alarms were genuine loan requests between partners, which shows exactly where the detector draws its line. In social engineering, the system flagged ordinary harmless conversations more often than the ones I built to look suspicious, a pattern I wouldn’t have seen without measuring false alarms separately.

4. Context isn’t always the advantage you expect. I also tested single messages cut from the conversations. Whole conversations did better for coercive control and self-harm, as expected, but bullying went the other way: the single message was detected more often than the full conversation. With only five items per category that shows direction, not proof, but it changed how I think about long conversations.

5. COVA taught me what to build in from the start. In COVA I had to build label-audit tooling after the fact, and it corrected 1,594 mislabeled conversations. This time I built the safeguards in before generating anything at scale:

• Pilot first, with fixed pass/fail checks. Each category ran a small pilot against balance, shortcut and diversity checks that were set in advance and couldn’t change after seeing the pilot. Pilot conversations never entered the test set.
• Freeze and fingerprint everything. Every file got a cryptographic fingerprint before the run, so any later change, accidental or not, would be detectable. The same discipline covered what I didn’t look at: Tuteliq’s harness also ran their own benchmark datasets in the same job, and I left those results unopened.
• Check for shortcuts in both directions. I listed phrases and topics that appear on only one side, so the report could disclose anything a detector might match instead of the harm.
• Get a second, blind reader. Dr. Roy checked a random sample without seeing my labels, and every disagreement was read and discussed.
• Report uncertainty, not just a score. Every rate comes with a 95% Wilson interval.

COVA was where I learned to build a dataset. This project was where I learned to make one I could defend.

6. Wording is part of the method. My test sets were about half harmful. On real platforms harm is rare, so the same detector would have much lower precision. A number without its scope can mislead, which is why both reports end with a claims register saying exactly how each figure may be quoted.

🔭 What I’d improve next time

• Write more harmless look-alikes for social engineering, the category where the line was hardest to draw.
• Generate conversations where hostility builds gradually, forcing conversations to take more ‘turns’, since real escalation is usually slower than generated dialogue.
• Make conversations read more like real text threads, a point Dr. Roy raised in his review: varied openings, no sudden topic changes, and replies that answer the message before them so the conversations ‘flow’ better and more ‘real’.
• Use new, unseen conversations for any retest, since Tuteliq now has access to this dataset.

📄 Read the reports

Both are on scamlingua.org.

Results are from synthetic, English-language text conversations (detector tuteliq-detect-v5.20261015, tested October 19, 2026). This is an independent evaluation, not a certification or audit.

🙏 Thank you to Gabriel Sabadin and the Tuteliq team for this rare opportunity and for welcoming an independent review, weaknesses and all. Few companies invite outside testing and then publish the gaps, and that made this work possible. Thank you also to Dr. Ayan Roy for his time as second evaluator. Looking forward to working with both you in the near future to protect children and the elderly from malicious actors and offenders.