Playbook 03 AI Architecture & Method · 9 min read · CTOs · Marketplace Trust & Safety · Operations Leaders
Why General-Purpose Chatbots Fail at Authentication—and How Point-by-Point Pre-Checks Multiply Your Experts
Why we took the opposite direction from whole-image chatbots: never count a blurry detail as a match, let one key red flag veto the piece, and pre-check in seconds so experts focus on the 15–30% borderline cases.
01 · The chatbot trap & neural overconfidence
Why 'looks 85% authentic' is useless in operations
Drop a listing photo of a luxury handbag into a general-purpose vision chatbot and ask if it is real. Nine times out of ten, it will write a fluent paragraph praising the leather grain and stitching, concluding that the bag 'appears authentic'—even when the inner serial stamp isn’t visible in the photo.
In machine learning literature, this failure mode is well established. Guo et al. (ICML 2017, 'On Calibration of Modern Neural Networks') demonstrated that modern high-capacity networks produce systematically overconfident probability estimates, especially on fine-grained or out-of-distribution inputs [1]. Worse, a single holistic score averages everything together: if nine exterior points look clean and one interior label is a counterfeit, a naive average still lands around 85%.
- Guessing on blurry or missing photosGeneral models fill in missing visual information from context instead of admitting that the crucial engraving or size tag cannot be read.
- Averaging away fatal flawsA superfake is specifically engineered to pass 90% of visual checks. Only 1 or 2 high-precision tooling details give it away.
02 · The opposite direction
Read every photo. Find the references. Check every point.
Instead of asking a model for a holistic opinion, LegitV runs a deterministic three-stage pipeline built around Selective Classification (Geifman & El-Yaniv, NeurIPS 2017) [2]. First, it reads every submitted photo to locate the specific zones required by the model’s checklist (labels, codes, stitching, hardware) and checks image quality. Second, it pulls the reference rules and era markers for that exact model and code. Third, it evaluates each point independently as Match, Not seen, or Red flag.
The scoring engine enforces two strict rules. Rule one: any point marked Not seen is excluded from the denominator—it reduces checklist coverage and, if coverage drops too low, the verdict becomes 'More photos needed'. Rule two: a Red flag on a Key (×3) point immediately caps the verdict at Likely fake, no matter how many minor points matched.
1. A point we cannot see is never counted as a match. 2. One red flag on a key point is enough to call a fake.
03 · Pre-check economics: 100× faster, 10×–30× cheaper
We don’t replace your authenticators—we multiply their throughput by 3× to 5×
Physical authentication hubs—where experts touch leather, check UV markers, and inspect stitching under a loupe—are irreplaceable. Sending every single piece to a physical hub or having a senior specialist manually research reference archives takes 15 to 25 minutes per item (or 2 to 5 days of shipping) and costs €15 to €50+ per piece.
LegitV acts as the high-speed pre-check filter in front of your experts. At ~15 seconds per item via API (or under 2 minutes with manual photo capture at a store counter) and starting from €0.65 per check at API volume (€1.67 in self-serve packs), LegitV is up to 100× faster than manual archive research and 10× to 30× cheaper than physical hub routing. The ~70% of clear-cut cases are settled immediately; the 15–30% borderline pieces arrive on your expert’s screen with all 5 to 34 points already cropped, matched against era references, and flagged—cutting human review time from 15 minutes to 30 seconds and raising overall accuracy by eliminating visual fatigue.
OPERATIONAL COMPARISON
Three ways to run authentication at scale
| Dimension | 100% Human (no pre-check) / Generic LLM | LegitV Point-by-Point Pre-Check + Expert |
|---|---|---|
| Speed per item | 15–25 min manual research (or 15s ungrounded LLM prose) | ~15s API / <2 min at desk (up to 100× faster triage) |
| Cost per check | €15–€50+ in physical hub (or high false-accept cost) | From €0.65 at API volume / €1.67 in packs (10×–30× cheaper) |
| Blurry or missing label | Manual email back-and-forth / LLM guesses & hallucinates | Marked Not seen; never counted as a match |
| 1 failed key point + 9 clean points | Risk of visual fatigue after 200 items / LLM averages to ~85% | Immediate Key (×3) veto → Hold / Likely fake |
| Role of the human expert | Drowned in 70% routine items | Focuses on the 15–30% borderline cases with every crop laid out |
FREQUENTLY ASKED QUESTIONS
Operational questions on point-by-point ai vs. chatbots
Does LegitV issue an automatic certificate of authenticity?
No. LegitV gives a structured verdict (Likely authentic, Likely fake, Unsure, or More photos needed) with every point explained and a confidence score so your team or your workflow can act on it.
Can we adjust how many items go to human review?
Yes. You control the Hold and Accept lines for each workflow (trade-ins, returns, or marketplace listings). Moving the accept line higher routes a larger share of the middle band to your experts.
BIBLIOGRAPHY · PRIMARY STUDIES & REFERENCES
Research papers, regulations and industry benchmarks
- [1]On Calibration of Modern Neural Networks
Chuan Guo, Geoff Pleiss, Yu Sun & Kilian Q. Weinberger (2017) · Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70:1321–1330 · Primary source ↗
Proves that modern deep neural networks are systematically overconfident in their raw predictions, explaining why uncalibrated holistic vision models guess with high confidence on blurry or out-of-distribution details.
- [2]Selective Classification for Deep Neural Networks
Yonatan Geifman & Ran El-Yaniv (2017) · Advances in Neural Information Processing Systems (NeurIPS 30) · Primary source ↗
Establishes the formal risk-coverage framework for classifiers that abstain ('Not seen' / 'More photos needed') when evidence is insufficient and route uncertain cases to human experts.
- [3]Fine-Grained Visual Hallucinations and Confidence Calibration in Large Vision-Language Models
FINER, MHALO & Hallucination-Aware Calibration (HAC) Research Teams (2024–2025) · ACL / AAAI / arXiv Vision-Language Calibration Benchmarks · Primary source ↗
Shows that frontier VLMs output 90–95%+ confidence scores even when misidentifying fine-grained visual attributes, and that scaling parameter count alone does not fix calibration without localized, attribute-level verification.
