Data Deduplicator

Pricing

Rating

Developer

Actor stats

Categories

Share

Data Deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows automatically with case-insensitive matching and whitespace trimming built in.

Copy to your AI assistant

Copy this block into ChatGPT, Claude, Cursor, or any LLM to start using this actor.

parsebird/dataset-deduplicator on Apify. Call:ApifyClient("TOKEN").actor("parsebird/dataset-deduplicator").call(run_input={...}), then client.dataset(run["defaultDatasetId"]).list_items().items for deduplicated results. Key inputs:datasetIds(array of strings — Apify dataset IDs to merge),jsonData(array of objects — direct JSON input, alternative to datasetIds),fields(array of strings, required — field names for dedup key). Matching is case-insensitive with whitespace trimming. First occurrence is kept. Full actor spec: fetch build via GEThttps://api.apify.com/v2/acts/parsebird~dataset-deduplicator(Bearer TOKEN). Get token: https://console.apify.com/account/integrations

What does Data Deduplicator do?

This Actor merges one or more Apify datasets and removes duplicate rows based on fields you specify. It's the fastest way to clean up scraped data before analysis or export.

• Single-field dedup — deduplicate by url, email, phone, or any single field
• Composite key dedup — combine multiple fields like firstName + lastName + company to identify unique records
• Smart matching — case-insensitive comparison with automatic whitespace trimming
• Multi-dataset merge — combine items from multiple dataset IDs before deduplication
• Direct JSON input — pass data directly as a JSON array instead of referencing datasets

How to use it (6 steps)

1. Run your scraper(s) — collect data into one or more Apify datasets
2. Copy the dataset ID(s) — find them in the Apify Console under your run's Storage tab
3. Choose your dedup fields — pick the field(s) that uniquely identify each record
4. Run this Actor — pass the dataset IDs and field names as input
5. Get clean data — deduplicated items appear in the output dataset

Input parameters

*Provide either datasetIds or jsonData (or both).

Composite key examples

Output example

Deduplicated items retain their original structure — no fields are added or removed:

[
{"name":"John Doe","email":"john@example.com","company":"Acme"},
{"name":"Jane Smith","email":"jane@example.com","company":"Beta"},
{"name":"Bob Wilson","email":"bob@example.com","company":"Gamma"}
]

A stats key is stored in the key-value store:

{
"totalLoaded":5000,
"uniqueKept":3200,
"duplicatesRemoved":1800,
"datasetsProcessed":3
}

How to use via API

Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("parsebird/dataset-deduplicator").call(run_input={
"datasetIds":["DATASET_ID_1","DATASET_ID_2"],
"fields":["email"],
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(f"Unique items: {len(items)}")

Node.js

import{ ApifyClient }from'apify-client';
const client =newApifyClient({token:'YOUR_API_TOKEN'});
const run =await client.actor('parsebird/dataset-deduplicator').call({
datasetIds:['DATASET_ID_1','DATASET_ID_2'],
fields:['firstName','lastName','company'],
});
const{ items }=await client.dataset(run.defaultDatasetId).listItems();
console.log(`Unique items: ${items.length}`);

cURL

curl-X POST "https://api.apify.com/v2/acts/parsebird~dataset-deduplicator/runs?token=YOUR_API_TOKEN"\
-H"Content-Type: application/json"\
-d'{
 "datasetIds": ["DATASET_ID_1"],
 "fields": ["url"]
 }'

Tips and best practices

• Start with a single field — url or email usually covers most use cases
• Use composite keys carefully — the more fields, the stricter the matching (fewer duplicates found)
• Matching is always case-insensitive with whitespace trimming — no configuration needed

Pricing

This Actor uses a pay-per-event pricing model.

Charged per 1,000 items loaded (not per unique item). Platform compute costs are additional.