Building Resilient Web Scraping Agents with Playwright and LLM Parser Fallbacks

1. The Fragility of DOM Selectors

Traditional web scrapers relying on hardcoded CSS selectors (e.g. div.product-card > span.price-tag) break whenever target websites update their frontend framework, re-generate Tailwind utility classes, or deploy new UI components. For automated business intelligence, pricing engines, and lead collection, fragile scrapers cause continuous pipeline downtime.

2. Self-Healing AI Scraper Architecture

At Kroma Code, we build self-healing scrapers using Playwright and LLM fallback locators. When a CSS selector fails to locate a target DOM node, the scraper captures a clean snapshot of the DOM tree, sends the semantic structural snippet to a fast LLM, and receives an updated selector dynamically.

3. Code Example: Self-Healing Selector in Python

from playwright.async_api import async_playwright
import httpx

async def extract_product_price(page, primary_selector: str):
    try:
        # 1. Attempt primary CSS selector execution
        element = await page.wait_for_selector(primary_selector, timeout=3000)
        return await element.inner_text()
    except Exception:
        print("[WARN] Primary selector failed. Triggering LLM Self-Healing Locator...")
        
        # 2. Capture sanitized DOM snippet
        body_html = await page.evaluate("document.body.innerText.slice(0, 3000)")
        
        # 3. Query LLM to identify target content
        async with httpx.AsyncClient() as client:
            res = await client.post("https://api.openai.com/v1/chat/completions", json={
                "model": "gpt-4o-mini",
                "messages": [{
                    "role": "user", 
                    "content": f"Extract price from text snippet: {body_html}"
                }]
            }, headers={"Authorization": "Bearer YOUR_SECRET_KEY"})
            
            return res.json()["choices"][0]["message"]["content"]
      

4. Production Impact

Self-healing scraping pipelines operate with over 99.9% uptime, eliminating manual code fixes when third-party sites update their markup.