{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidclpylb3gmb3ddv7ntkg75joacbgagkvsm6umlqib2gigrzoi36m",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mommxlvf7b52"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreiaykgvjrd7vr3d2fgdeqcnrban374t6us5gowid5fzirubnni5lfy"
    },
    "mimeType": "image/webp",
    "size": 84204
  },
  "path": "/parastejpal987cmyk/scraping-dynamic-web-pages-without-selectors-using-ai-vision-typescriptjavascript-tutorial-1n63",
  "publishedAt": "2026-06-19T05:29:11.000Z",
  "site": "https://dev.to",
  "tags": [
    "webscraping",
    "javascript",
    "ai",
    "node",
    "RapidAPI Opticparse Listing"
  ],
  "textContent": "****# Scraping Dynamic Web Pages Without Selectors Using AI Vision (TypeScript/JavaScript Tutorial)\n\nWeb scraping has traditionally been a game of cat-and-mouse. You spend hours writing fine-tuned CSS selectors or XPath paths, only for the website to change its layout or class names (especially on modern frameworks with generated CSS class names like `css-1ux802d`), breaking your entire data pipeline overnight.\n\nIn this tutorial, we will learn how to build a **selector-free scraper** using **Opticparse** , an AI-powered scraping tool that captures webpage screenshots and uses Gemini's multimodal vision intelligence to extract structured JSON data.\n\nWe will use the official **Opticparse JavaScript/TypeScript SDK** to extract data in less than 10 lines of code.\n\n##  The Concept: AI Vision Scraping\n\nInstead of parsing HTML source code directly, Opticparse:\n\n  1. Launches a headless Chromium instance using Playwright.\n  2. Navigates to the target page and takes a full-page snapshot.\n  3. Passes the screenshot to an AI Vision Agent (Gemini) along with a text prompt.\n  4. Returns clean, parsed JSON matching your description.\n\n\n\nBecause it mimics how a real human looks at the page, it does not care about dynamic CSS class name changes, shadow DOMs, or obfuscated HTML.\n\n##  Setup & Installation\n\nInstall the official client library:\n\n\n\n    npm install opticparse-js\n\n\n###  Get Your API Key\n\nYou can get an API key in two ways:\n\n  1. **RapidAPI Hub** : Access the API globally on the RapidAPI Opticparse Listing. Subscribe to the Free basic tier to get a RapidAPI Key.\n  2. **Private Host** : If you hosted the Docker microservice container yourself (e.g. on Render), use your private `OPTICPARSE_API_KEY`.\n\n\n\n##  Code Example: Scraping Hacker News\n\nLet's say we want to scrape the top 5 articles, their link URLs, and score points from the homepage of Hacker News.\n\nHere is how you do it:\n\n\n\n    import { OpticparseClient } from 'opticparse-js';\n\n    // Initialize the client.\n    // If using the RapidAPI marketplace, set useRapidApi: true\n    const client = new OpticparseClient({\n      apiKey: 'YOUR_RAPIDAPI_KEY_HERE',\n      useRapidApi: true\n    });\n\n    async function runScrape() {\n      console.log('Scraping Hacker News articles...');\n\n      try {\n        const data = await client.scrape({\n          targetUrl: 'https://news.ycombinator.com',\n          extractionQuery: 'Extract the top 5 article titles, their link URLs, and score points as a JSON list of objects.',\n          viewportWidth: 1280,\n          viewportHeight: 1000\n        });\n\n        console.log('Scraped Data Output:');\n        console.log(JSON.stringify(data, null, 2));\n\n      } catch (error) {\n        console.error('Scraping failed:', error);\n      }\n    }\n\n    runScrape();\n\n\n###  Sample Output\n\nThe client will automatically handle the asynchronous execution, image loading, and return a clean, fully-typed JSON structure:\n\n\n\n    [\n      {\n        \"title\": \"Why I still use Vim\",\n        \"url\": \"https://example.com/vim\",\n        \"points\": 142\n      },\n      {\n        \"title\": \"Show HN: Opticparse - AI Visual Scraper\",\n        \"url\": \"https://github.com/parastejpal987-cmyk/opticparse\",\n        \"points\": 98\n      }\n    ]\n\n\n##  Advanced Options\n\nThe SDK client supports configuring the browser environment to handle dynamic loading states:\n\n\n    typescript\n    const result = await client.scrape({\n      targetUrl: 'https://example.com',\n      extractionQuery: 'Extract details...',\n\n      // Custom screen sizes for responsive layouts\n      viewportWidth: 1920,\n      viewportHeight: 1080,\n\n      // Wait until page is completely loaded ('networkidle' | 'load' | 'domcontentloaded')\n      waitUntil: 'networkidle',\n\n      // Adjust timeout threshold (in milliseconds) for slower connec\n",
  "title": "Scraping Dynamic Web Pages Without Selectors Using AI Vision (TypeScript/JavaScript Tutorial)"
}