****# Scraping Dynamic Web Pages Without Selectors Using AI Vision (TypeScript/JavaScript Tutorial)

Web scraping has traditionally been a game of cat-and-mouse. You spend hours writing fine-tuned CSS selectors or XPath paths, only for the website to change its layout or class names (especially on modern frameworks with generated CSS class names like css-1ux802d), breaking your entire data pipeline overnight.

In this tutorial, we will learn how to build a selector-free scraper using Opticparse, an AI-powered scraping tool that captures webpage screenshots and uses Gemini's multimodal vision intelligence to extract structured JSON data.

We will use the official Opticparse JavaScript/TypeScript SDK to extract data in less than 10 lines of code.

The Concept: AI Vision Scraping

Instead of parsing HTML source code directly, Opticparse:

1. Launches a headless Chromium instance using Playwright.
2. Navigates to the target page and takes a full-page snapshot.
3. Passes the screenshot to an AI Vision Agent (Gemini) along with a text prompt.
4. Returns clean, parsed JSON matching your description.

Because it mimics how a real human looks at the page, it does not care about dynamic CSS class name changes, shadow DOMs, or obfuscated HTML.

Setup & Installation

Install the official client library:

npm install opticparse-js

Get Your API Key

You can get an API key in two ways:

1. RapidAPI Hub: Access the API globally on the RapidAPI Opticparse Listing. Subscribe to the Free basic tier to get a RapidAPI Key.
2. Private Host: If you hosted the Docker microservice container yourself (e.g. on Render), use your private OPTICPARSE_API_KEY.

Code Example: Scraping Hacker News

Let's say we want to scrape the top 5 articles, their link URLs, and score points from the homepage of Hacker News.

Here is how you do it:

import { OpticparseClient } from 'opticparse-js';

// Initialize the client. 
// If using the RapidAPI marketplace, set useRapidApi: true
const client = new OpticparseClient({
 apiKey: 'YOUR_RAPIDAPI_KEY_HERE',
 useRapidApi: true
});

async function runScrape() {
 console.log('Scraping Hacker News articles...');

 try {
 const data = await client.scrape({
 targetUrl: 'https://news.ycombinator.com',
 extractionQuery: 'Extract the top 5 article titles, their link URLs, and score points as a JSON list of objects.',
 viewportWidth: 1280,
 viewportHeight: 1000
 });

 console.log('Scraped Data Output:');
 console.log(JSON.stringify(data, null, 2));

 } catch (error) {
 console.error('Scraping failed:', error);
 }
}

runScrape();

Sample Output

The client will automatically handle the asynchronous execution, image loading, and return a clean, fully-typed JSON structure:

[{"title":"Why I still use Vim","url":"https://example.com/vim","points":142},{"title":"Show HN: Opticparse - AI Visual Scraper","url":"https://github.com/parastejpal987-cmyk/opticparse","points":98}]

Advanced Options

The SDK client supports configuring the browser environment to handle dynamic loading states:

typescript
const result = await client.scrape({
 targetUrl: 'https://example.com',
 extractionQuery: 'Extract details...',

 // Custom screen sizes for responsive layouts
 viewportWidth: 1920,
 viewportHeight: 1080,

 // Wait until page is completely loaded ('networkidle' | 'load' | 'domcontentloaded')
 waitUntil: 'networkidle',

 // Adjust timeout threshold (in milliseconds) for slower connec