Pricing

Rating

Developer

Actor stats

Categories

Share

Gemini Scraper — Apify Actor

Scrapes text responses from Google Gemini by replaying authenticated browser requests. Accepts a list of queries, sends each to Gemini, and saves the full AI-generated response to an Apify dataset.

⚠️ This actor uses Gemini's internal browser API, not the official Gemini REST API. Session cookies are hardcoded in the source and must be refreshed in the code when they expire (~24 hours).

Features

• Send multiple queries in a single run
• Only input required: a list of queries
• Full response text extracted from Gemini's streaming protocol
• Per-query error isolation — one failure does not stop the rest
• Dataset output with structured fields: query, response, error, scrapedAt

Input

Only one field is required.

Example Input JSON

{
"queries":[
"best smartphones 2026",
"python vs javascript for web development",
"how does quantum computing work"
]
}

Output

Each query produces one record in the default dataset.

Success Record

{
"query":"best smartphones 2026",
"response":"The top smartphones in 2026 include...",
"error":null,
"scrapedAt":"2026-06-19T10:30:00.000Z"
}

Failure Record

{
"query":"best smartphones 2026",
"response":null,
"error":"HTTPError: 403 Forbidden",
"scrapedAt":"2026-06-19T10:30:00.000Z"
}

When Cookies Expire

All session values (cookies, headers, params) are hardcoded at the top of src/main.py under COOKIES, HEADERS, PARAMS, and SESSION_ID.

When requests start returning 401 or 403, refresh the values:

Step-by-step

1. Open https://gemini.google.com in Chrome while signed in.
2. Press F12 → Network tab.
3. Send any message in Gemini.
4. Filter by StreamGenerate → click the request.
5. Go to Headers → Request Headers → copy the Cookie header.
6. Extract __Secure-STRP, NID, and COMPASS values.
7. Update COOKIES in src/main.py.
8. From the same request's Query String Parameters, copy bl and f.sid.
9. Update PARAMS in src/main.py.
10. Redeploy the actor.

Project Structure

gemini-actor/
├── src/
│ └── main.py # Core actor logic + hardcoded session values
├── .actor/
│ ├── actor.json # Apify actor manifest
│ └── INPUT_SCHEMA.json # Input schema(queries only)
├── output_schema.json # Dataset view columns
├── dataset_schema.json # Per-record field types
├── requirements.txt # Python dependencies
├── Dockerfile # Container build
├── Agent.md # Internal developer/agent guide
├── claude.md # AI assistant context file
└── README.md # This file

Local Development

# Install dependencies
pip install apify requests
# Run locally (requires Apify CLI)
apify run

Dependencies

apify>=2.0.0
requests>=2.31.0

Limitations

• Session cookies expire in ~24 hours and require a manual code update + redeploy
• Subject to Gemini's internal rate limits
• Response format may break if Google updates the StreamGenerate protocol
• Not suitable for high-volume production use without cookie rotation

License

MIT