Newshot

Tiny AI Security Model Beats Five Frontier LLMs on Privacy Detection

Original text(via HN AI)

ProofAgent Nano, now in three languages.

题图
题图

ProofAgent Nano comes in two versions. Both are 98M-parameter models that run on device, on CPU or GPU, so no text leaves the machine.

ProofAgent Nano

The English model. #1 on the full English PIIMB test set, ahead of the 36 published models in the comparison and of five frontier LLMs. The detailed results are below.

ProofAgent Nano Multilingual

AI privacy and safety detection in English, French and Arabic, trained natively in all three. #1 of 37 on the PIIMB leaderboard.

ProofAgent Nano Multilingual screens prompts, documents, tool outputs and model responses before they reach an LLM or an AI agent. In one pass it finds personal data, down to the exact words, and flags AI risks such as prompt injection, jailbreaks and data exfiltration.

Personal data, AI security and safety, in one pass.

AreaExamples
Personal data (PII)Names, emails, phone numbers, addresses and locations, national IDs and SSNs, payment cards, IBANs and bank details, health data, credentials and secrets
AI securityDirect and indirect prompt injection, jailbreaks, system-prompt extraction, data exfiltration, tool misuse, obfuscated payloads
Safety and complianceIllegal activity, harmful content, fraud and scams, risky regulated advice

Each piece of personal data is tagged with its exact position and type, for example PERSON/FIRST_NAME, GOV_ID/NATIONAL_ID or BANK_FINANCIAL/IBAN, so it can be masked, logged or blocked.

Real model output in English, French and Arabic.

Every example below is real output from ProofAgent Nano Multilingual. Scores run from 0 (clearly safe) to 1 (clearly a risk). Each risk has its own blocking threshold.

English

SarahPERSON/FIRST_NAME
MitchellPERSON/LAST_NAME
4539 1488 0343 6467PAYMENT_CARD/CREDIT_CARD
[email protected]CONTACT/EMAIL
+1 415 555 0132CONTACT/PHONE

Français

JulienPERSON/FIRST_NAME
MoreauPERSON/LAST_NAME
1 85 05 78 006 084 36GOV_ID/SSN
FR76 3000 6000 0112 3456 7890 189BANK_FINANCIAL/IBAN

العربية · Arabic

أحمدPERSON/FIRST_NAME
بن سالمPERSON/MIDDLE_NAME
العتيبيPERSON/LAST_NAME
1087654321GOV_ID/NATIONAL_ID
0551234567CONTACT/PHONE
حي النرجسLOCATION_COARSE/REGION
بالرياضLOCATION_COARSE/CITY

AI is moving from answering questions to taking action.

The first wave of enterprise AI answered questions. The current wave acts. LLMs and AI agents now read company knowledge, work through customer records, retrieve documents and call connected tools on behalf of employees and customers.

Traditional applications

Store and process data according to predefined workflows. The fields, the flows and the people who can reach them are known in advance.

Agentic AI

Interprets unstructured content, reasons over it, calls tools and takes actions. Every step is a new place where sensitive data can surface or untrusted instructions can enter:

  • Prompts and model responses
  • Retrieved documents and web pages
  • Agent memory
  • Tool inputs and outputs

Identity, network and data-security controls remain necessary. They decide who can reach a system and where data may travel. They do not always read what an AI interaction actually says: whether a prompt carries a customer's health record, or whether a retrieved page hides an instruction telling the agent to export its data.

That semantic layer of inspection is what ProofAgent Nano provides.

Why local AI protection matters.

Inspect sensitive data where it lives

Sensitive information can be inspected locally or within enterprise-controlled infrastructure before anything is forwarded to an external AI service. The text being checked does not need to travel to another provider to be checked.

Make repeated screening practical

Specialized, CPU-capable detection can screen every prompt, response and agent step without a frontier-LLM API call for each inspection. Hardware, electricity, maintenance and integration still carry costs; what changes is that inspection volume no longer scales per-call API spend.

Turn findings into controls

Structured detection findings can drive masking, policy enforcement, escalation, audit trails and compliance review, so privacy and safety decisions are recorded rather than left to each application.

A leading privacy result from a compact specialist model.

On the full English PIIMB test set, ProofAgent Nano reaches 0.935 average F2, ahead of the 36 published models in the comparison.

PIIMB full test set · 121,409 sentences

What it is. PIIMB, the PII Masking Benchmark, is a public test of how well a model finds personal information, such as names, addresses, ID numbers and bank details, so it can be hidden. It uses 121,409 English sentences, and published results cover 36 models, including privacy tools from OpenAI, NVIDIA and Microsoft.

Why it matters. Personal data must be masked before an AI system reads emails, records or contracts. Missing it exposes data; hiding too much makes the text useless. PIIMB scores that balance on the same sentences for every model.

Average F2, selected published models

OpenMed's 1.4B-parameter filter is the strongest published model at 0.918. Nano's lead is largest on the hardest task, privy.

Model size versus privacy accuracy

A task-specialized model can reach high privacy-detection accuracy without the scale of much larger models. Nano scores above 1.4B-parameter models while being about 14× smaller by parameter count.

Published PIIMB scores; Nano's score computed locally with the official scorer. Parameter count alone does not determine runtime, memory use or deployment cost. This is a privacy-masking comparison, not a general AI safety ranking.

Head-to-head with five frontier LLMs · same 2,000 sentences

All six systems were evaluated on the same stratified 2,000-sentence PIIMB sample (500 per task) with the same PIIMB scorer. The frontier LLMs ran zero-shot with a detailed prompt listing every PII type; Nano is a specialized model trained for this task. These scores are a separate experiment and are not merged with the full-set comparison.

Average F2 · 2,000-sentence sample

Precision, recall and per-task F2

ModelPRnemotronprivy
ProofAgent Nano0.9290.9350.8980.939
Claude Fable 5.10.8820.9100.7620.899
Claude Opus 5.50.8850.8940.7670.867
GPT-6.1 Sol0.8340.8810.7290.835
GPT-6 Astra0.8350.8800.7240.837
Claude Haiku 5.50.8550.8270.6740.795

The strongest frontier models are excellent on simple prose: Fable 5.1 and Opus 5.5 lead on ai4privacy. On enterprise business records (nemotron-pii) every frontier LLM scores 0.67–0.77, against 0.898 for Nano.

Measured throughput on one laptop

Measured on an Apple M5 Max, batched, over the 2,000-sentence sample (84,414 tokens). These are throughput figures, not single-request latency, and were recorded while another job shared the machine.

What it catches, and what a frontier model missed.

Three selected cases from the evaluations above, shown as recorded. They illustrate behaviour; they do not establish comprehensive security superiority.

Protect the borrower, keep the loan readable

Why it matters. Over-redaction makes documents unusable for downstream operations such as underwriting or servicing; under-redaction exposes sensitive data. Precision is what keeps masking deployable.

A hidden instruction that tries to export every customer ticket

Why it matters. RAG pipelines and autonomous agents consume untrusted external content. An instruction hidden in a page a human never sees can still reach the model, redirect it and request customer data.

A fake "assistant" instruction asking for everyone's home address

Why it matters. Recognising an attack is half the signal. Knowing its intended consequence (here, a bulk export of personal addresses) is what lets a policy decide whether to block, mask or escalate.

These are selected cases. Across the comparison, frontier LLMs reliably catch classic identifiers such as names, SSNs and emails. In the 70-record AI-risk pilot, Nano caught slightly more injection and exfiltration cases (22 vs 21 exfiltration, 24 vs 22 indirect injection) with fewer exfiltration false alarms (1 vs 6), while Fable was better at inferring sensitive traits. The pilot is separate from both PIIMB evaluations.

One compact model. Multiple layers of AI protection.

Nano produces detection signals. The application or platform around it decides whether to mask, block, escalate or allow.

ApplicationWhat ProofAgent Nano doesWhere it operatesBusiness value
PII detection and maskingIdentifies sensitive information at precise character spans so it can be masked.Prompts, responses, documents, logsUse AI on real records while keeping identifiers out of model inputs and outputs.
Input and output AI guardrailsFlags potential risks in prompts before processing and in responses before delivery.LLM and agent gatewaysOne consistent inspection point across every assistant and model provider.
Data-loss preventionIdentifies sensitive identifiers and flags potential disclosure or exfiltration requests.Enterprise assistants, messaging, workflowsReduces the chance that AI interactions leak customer or employee data.
Prompt injection and jailbreak detectionFlags potential instruction overrides, jailbreaks and system-prompt extraction attempts.User input, retrieved contentGives security teams a signal on attempts to redirect AI systems.
Agent and RAG protectionInspects retrieved documents and tool outputs for sensitive content and suspicious instructions.RAG pipelines, agent tool callsLets agents consume external content with an inspection step in between.
Child safety and chat protectionFlags potential personal-data solicitation, disclosure and off-platform contact patterns.Chat and community productsSupports moderation teams protecting younger users.
Data hygiene and dataset redactionIdentifies and masks sensitive information in stored and training data.Logs, tickets, transcripts, datasetsCleaner data for analytics and model training, with less manual review.
Compliance intelligenceAssociates each finding with relevant provisions in 38 frameworks to support policy review.Policy, audit and review workflowsAudit trails that record why content was masked or flagged.

Detection is not enforcement. Nano reports what it finds; masking, blocking and escalation policies are set by the application. Detecting a risk does not establish legal compliance.

Regulation-aware risk intelligence.

Every finding can carry the regulatory provisions it relates to, across 38 regulatory and standards frameworks. Compliance and audit teams see not only what was masked or flagged, but which rule made it relevant. The mapping is versioned and can be updated without retraining the model.

Structured findings support governance and audit workflows. The mapping is informational, is not legal advice, and does not by itself establish compliance or confer certification.

From AI detection to enterprise AI governance.

ProofAgent Nano is the local detection layer inside ProofAgent's AI governance infrastructure. The model detects; the platform around it evaluates, orchestrates and records.

The opportunity behind a compact protection layer.

Organizations are deploying LLMs and AI agents across workflows that touch sensitive data. Each deployment needs a way to see what those systems handle.

Inspect sensitive content without sharing it with yet another external provider.

AI interactions repeat across many applications, users and autonomous agent steps; inspection has to keep pace.

Compact specialized models open options for enterprise-controlled infrastructure and CPU-based inference.

Structured findings become inputs to enterprise security, audit and policy workflows.

The reported benchmark shows a specialized 98M model outperforming much larger systems on a defined task.

Building the next generation of trusted AI.

ProofAgent Nano represents ProofAgent's vision for making privacy, safety, and governance part of the AI lifecycle.

As AI systems become more capable and autonomous, organizations need protection that operates closer to their data, applications, and decisions.

By combining specialized detection, local inference, and structured policy intelligence, ProofAgent aims to make advanced AI protection practical across enterprise environments.

Work with us on ProofAgent Nano.

Pilots, independent validation, integrations and partnerships.

Methodology: the full-set PIIMB result was computed locally with the official PIIMB evaluation tool against published scores and has not been submitted to the public leaderboard. The 2,000-sentence frontier-LLM comparison and the 70-record AI-risk pilot are separate experiments; their results are not combined. English only has been evaluated. ProofAgent Nano is proprietary; this announcement does not disclose model architecture, training recipes, dataset composition or calibration details.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗