Skip to content
ShinyQ HQ: Back to HQ

← Back to the Labs

L3:fraud-review-platform · AI Wing · Hero

AI Fraud Document Review

OCR plus a two-pass LLM review of procurement packets, with a risk engine that had to pass its own gates.

Role
Creator and sole engineer: designed, directed, reviewed and shipped it with AI coding agents
Period
May 2026 to Oct 2026

Results

  • 90.9% to 36.4%

    False-positive rate on noisy clean packets

    Offline evaluation on a development gold set.

    Verified

  • 76.2% to 0%

    Fail-open rate

    Offline evaluation on the same development gold set.

    Verified

  • 0 of 3,780

    Production runs where the shadow engine changed user-visible output

    The hybrid engine cut false positives to 9.1% offline, but recall fell to 0.841, so it failed its gate and ran in shadow mode only.

    Verified

  • 11.1% to 3.7%

    Prompt-injection success rate

    Live evaluation runs after a model change (false positives also fell from 26.1% to 21.7%); evaluation results, not production outcomes.

    Verified

Problem

Procurement teams need fast, trustworthy fraud review of document packets. Early versions raised alarms on clean packets, which erodes trust faster than a missed flag.

What I did

  • Built a two-pass pipeline (per-document findings, then a cross-document verdict) over OCR output, with live progress streaming.
  • Moved prompts into declarative YAML agents on Microsoft Agent Framework.
  • Built gold sets and CI gates for false-positive rate, recall and fail-open, then replaced a majority-vote guardrail and over-broad safe patterns.
  • Designed a hybrid rules plus LLM risk engine (rules catalog, severity floors, evidence verifier, duplicate fingerprints) and kept it in shadow mode when it failed the recall gate.
  • Handled a secret-exposure incident with a history purge, a zero-downtime two-key rotation and a move to Key Vault references.

Architecture

  1. HumanProcurement reviewerUploads a document packet
  2. ClientNext.js web appUpload, live progress, reports
  3. ServiceFastAPI backendTwo-pass pipeline, live progress
  4. AIDocument IntelligenceOCR: Read, then Layout
  5. AIAgent Framework YAML agentsPer-document, then cross-document
  6. AIAzure OpenAIInputs screened by Prompt Shields
  7. ServiceHybrid risk engineRules plus LLM, shadow mode
  8. DataCosmos DB and BlobHistory and reports

Data flow

  • Procurement reviewer to Next.js web app (document packet)
  • Next.js web app to FastAPI backend (REST, live progress)
  • FastAPI backend to Document Intelligence (OCR)
  • FastAPI backend to Agent Framework YAML agents (findings, verdict)
  • Agent Framework YAML agents to Azure OpenAI (model calls)
  • FastAPI backend sends asynchronously to Hybrid risk engine (shadow run)
  • Hybrid risk engine sends asynchronously to Cosmos DB and Blob (shadow results)
  • FastAPI backend to Cosmos DB and Blob (history, reports)

Stack

  • Next.js
  • FastAPI
  • Azure OpenAI
  • Document Intelligence
  • Microsoft Agent Framework
  • Azure AI Content Safety
  • Cosmos DB
  • Blob Storage
  • App Service
  • Bicep
  • GitHub Actions
  • Azure Managed Applications

L2 · On the journeyTechnical Consultant, Software & AI · Metrodata (PT Mitra Integrasi Informatika)