Autonomous Document Parsing: Extracting Complex Invoices and Contracts with Vision LLMs

1. The Limitations of Legacy Optical Character Recognition (OCR)

Legacy OCR tools (e.g. Tesseract or basic PDF parsers) rely on rigid bounding box coordinate rules. When an inbound invoice scan is rotated 5 degrees, contains hand-written notes, or uses multi-column table layouts, traditional OCR fails completely, outputting scrambled text lines that break downstream database ingestors.

2. Vision-First Extraction Architecture

By rendering PDF pages into high-resolution PNG images and passing them directly to Vision-capable LLMs (e.g. GPT-4o Vision or Claude 3.5 Sonnet Vision) alongside strict JSON Schema parameters, document processing becomes 100% layout-agnostic.

3. Code Example: Vision Invoice Extractor in Python

import base64
import httpx

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

async def parse_invoice_image(image_path: str):
    base64_image = encode_image(image_path)
    
    prompt = "Extract invoice JSON: {invoice_number, vendor_name, total_amount, line_items: [{description, price}]}"
    
    async with httpx.AsyncClient() as client:
        res = await client.post("https://api.openai.com/v1/chat/completions", json={
            "model": "gpt-4o",
            "messages": [{
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
                ]
            }],
            "response_format": {"type": "json_object"}
        }, headers={"Authorization": "Bearer YOUR_SECRET_KEY"})
        
        return res.json()["choices"][0]["message"]["content"]
      

4. Benefits

Vision-first parsing achieves **99.4% extraction accuracy** across multi-lingual invoices, crumpled receipts, and legal contract scans.