Pricing

Rating

Developer

Actor stats

Categories

Share

Sahibinden.com Web Scraper Documentation

Overview

This document provides comprehensive documentation for the Sahibinden.com web scraper built with Apify. The scraper is designed to extract car listings from Sahibinden.com while bypassing anti-bot measures, and store the data in a structured format in BaseRow to power an AI chatbot for used car price estimation.

Architecture

The solution consists of the following components:

1. Apify Actor - A cloud-based web scraper that handles the extraction of car listings from Sahibinden.com
2. Playwright with Stealth - Browser automation with anti-detection capabilities to bypass Cloudflare protection
3. Residential Proxies - IP rotation to avoid blocking and simulate real user traffic
4. BaseRow Integration - Data storage in a structured format for easy querying by the AI chatbot
5. Scheduler - Automated regular scraping to keep data fresh

Technical Implementation

1. Anti-Bot Bypassing Techniques

The scraper implements several techniques to bypass Sahibinden.com's Cloudflare protection:

• Residential Proxies: Uses Apify's RESIDENTIAL proxy group with Turkey country code to appear as legitimate Turkish users
• Browser Fingerprinting: Implements realistic browser fingerprinting to avoid detection
• Stealth Plugin: Uses playwright-stealth to modify browser behavior and evade detection
• Human-like Behavior: Adds random delays, realistic mouse movements, and proper request headers
• Non-headless Mode: Runs browser in non-headless mode for better Cloudflare bypass
• Cookie Management: Sets and maintains cookies to simulate returning users
• User-Agent Rotation: Rotates between realistic user agents for each request

2. Data Extraction

The scraper extracts comprehensive data from car listings including:

• Basic information (ID, URL, title, price, location)
• Vehicle specifications (make, model, year, fuel type, etc.)
• Detailed attributes (interior/exterior features, safety features, etc.)
• Technical specifications
• Images
• Seller information
• Description and condition details

The data extraction is implemented in two main functions:

• handleCategoryPage() - Extracts listing URLs from category pages and handles pagination
• handleDetailPage() - Extracts comprehensive data from individual car listing pages

3. BaseRow Integration

The BaseRow integration provides:

• Automatic storage of scraped data in a structured format
• Duplicate detection and handling
• Field mapping between scraped data and BaseRow table structure
• Batch processing for efficient data storage

4. Automation

The scraper is configured for automatic scheduling with:

• Configurable run frequency (hourly, daily, weekly)
• Performance monitoring
• Error handling and retry mechanisms
• Rate limiting to avoid overloading the target site

Setup Instructions

Prerequisites

• Apify account with access to residential proxies
• BaseRow account with a table set up for car listings
• Node.js 16+ for local development (optional)

Apify Actor Setup

1. Create a new Actor in your Apify account
2. Upload the code from this repository to the Actor
3. Configure Actor settings:
   • Memory: Minimum 4 GB recommended
   • Timeout: At least 30 minutes
   • Environment variables: None required

BaseRow Setup

1. Create a new table in BaseRow with the following structure:

2. Get your BaseRow API credentials:
   • API Token
   • Table ID
   • Database ID

Running the Scraper

Configuration Options

The Actor accepts the following input parameters:

{
"startUrls":["https://www.sahibinden.com/kategori/vasita"],
"maxConcurrency":1,
"maxRequestsPerCrawl":1000,
"proxyConfiguration":{
"useApifyProxy":true,
"apifyProxyGroups":["RESIDENTIAL"],
"countryCode":"TR"
},
"baseRowApiToken":"YOUR_BASEROW_API_TOKEN",
"baseRowTableId":"YOUR_BASEROW_TABLE_ID",
"baseRowDatabaseId":"YOUR_BASEROW_DATABASE_ID",
"scheduleInterval":"daily",
"startTime":"02:00"
}

Scheduling

To set up automatic scheduling:

1. Create a new Schedule in your Apify account
2. Set the desired frequency (recommended: daily)
3. Configure the Actor input as shown above
4. Enable the Schedule

AI Chatbot Integration

The data stored in BaseRow can be used to power an AI chatbot for used car price estimation. The chatbot can:

1. Take user input describing a car (e.g., "2017 Passat 3 parça boya 150bin km")
2. Query the BaseRow table for comparable listings
3. Calculate an estimated price range based on similar vehicles
4. Adjust the estimate based on factors like damage status, mileage, and year

Query Examples

To find comparable listings in BaseRow, use queries like:

SELECT*FROM cars 
WHERE make ='Volkswagen'
AND model ='Passat'
ANDyear='2017'
AND mileage BETWEEN130000AND170000

Troubleshooting

Common Issues

1. 403 Forbidden Errors

   • Check that residential proxies are properly configured
   • Verify that the stealth plugin is working correctly
   • Try reducing concurrency to 1
   • Increase delays between requests

2. Data Extraction Issues

   • Check if Sahibinden.com has changed their page structure
   • Update the selectors in the data extraction functions
   • Verify that the page is fully loaded before extraction

3. BaseRow Integration Issues

   • Verify API credentials are correct
   • Check that the table structure matches the expected fields
   • Look for API rate limiting issues

Maintenance

To keep the scraper running smoothly:

1. Regular Monitoring: Check the Actor runs for any errors or performance issues
2. Updates: Update the code if Sahibinden.com changes their website structure
3. Proxy Management: Ensure you have sufficient residential proxy credits
4. Data Cleaning: Periodically clean old or irrelevant listings from BaseRow

Limitations

• The scraper is designed specifically for Sahibinden.com and may not work on other websites
• Cloudflare protection methods may change, requiring updates to the bypassing techniques
• Very high volume scraping may still trigger rate limiting despite all precautions
• Some listings may have incomplete data if the seller didn't provide all information