RAG Pipeline

Pricing

Rating

Developer

Actor stats

Categories

Share

One-click RAG pipeline on Apify. Chunk text, generate embeddings, and store vectors in Pinecone or Qdrant -- all in a single actor run. MCP-ready for AI agent integration.

What It Does

This actor orchestrates three sub-actors in sequence to build a complete RAG (Retrieval-Augmented Generation) pipeline. Feed it your content and it handles chunking, embedding, and vector storage automatically. Returns a pipeline summary -- ready for orchestration or consumption by AI agents via MCP.

Your content
->RAG Content Chunker(chunk by paragraphs, sentences, or Markdown headers)
->RAG Embedding Generator(OpenAI or Cohere embeddings)
->RAG Vector Store Writer(upsert to Pinecone or Qdrant)

You provide your content, API keys, and vector DB config. The pipeline handles dataset handoff between steps automatically.

🆕 New Feature: Bulk File Upload

Already have a document? Upload it directly to Apify Storage and run the full pipeline against it — no crawling, no copy-pasting.

How to upload a file:

1. Go to Apify Console → Storage → Key-Value Stores
2. Click + Create new store → give it a name
3. Click + Add record → upload your .txt, .md, or .pdf file
4. Find your file → click the 🔗 icon to copy the direct URL

   Make sure the URL starts with api.apify.com — not console.apify.com. The console URL is the web page, not the file.

5. Paste the URL into the file_url field along with your API keys and run

Example — full pipeline from a file:

{
"file_url":"https://api.apify.com/v2/key-value-stores/YOUR_STORE_ID/records/document.md",
"chunking_strategy":"markdown",
"chunk_size":512,
"chunk_overlap":64,
"embedding_api_key":"sk-...",
"embedding_provider":"openai",
"embedding_model":"text-embedding-3-small",
"vector_db_api_key":"your-qdrant-key",
"vector_db_provider":"qdrant",
"index_name":"my-rag-collection",
"qdrant_url":"https://your-cluster.us-west-1.aws.cloud.qdrant.io:6333"
}

Or skip storage entirely — paste text or Markdown directly into the text field:

{
"text":"# My Document\n\nThis is my content...",
"chunking_strategy":"markdown",
"chunk_size":512,
"embedding_api_key":"sk-...",
 ...
}

Supported file formats: .txt, .md, .markdown, .html, .pdf Max file size: 5MB URL requirements: Must be a public HTTPS URL. Apify Storage, S3, Dropbox (shared public link), or GitHub raw URLs all work.

PDF note: Text-based PDFs are supported. Scanned/image-only PDFs have no text layer and will fail — convert them with OCR first. Office documents (.docx, .xlsx) are not yet supported.

Input

Content Source (choose one)

Option A — Direct text:

{"text":"# My Document\n\nContent..."}

Option B — Single file URL (Apify Storage):

{
"file_url":"https://api.apify.com/v2/key-value-stores/STORE_ID/records/doc.txt",
"chunking_strategy":"markdown",
"chunk_size":512
}

Option C — Multiple file URLs, bulk (Apify Storage):

{
"file_urls":[
"https://api.apify.com/v2/key-value-stores/STORE_ID/records/doc1.txt",
"https://api.apify.com/v2/key-value-stores/STORE_ID/records/doc2.md"
]
}

Option D — Dataset from crawler:

{"source_dataset_id":"your-crawler-dataset-id","source_dataset_field":"markdown"}

Priority order when multiple sources are provided: source_dataset_id > file_urls > file_url > text

Output

A single summary item in the default dataset:

{
"_summary":true,
"pipeline":{
"total_duration_seconds":12.345,
"steps":{
"chunker":{"actor":"labrat011/rag-content-chunker","status":"SUCCEEDED","duration_seconds":3.2},
"embedder":{"actor":"labrat011/rag-embedding-generator","status":"SUCCEEDED","duration_seconds":5.1},
"writer":{"actor":"labrat011/rag-vector-store-writer","status":"SUCCEEDED","duration_seconds":4.0}
}
},
"result":{
"total_upserted":42,
"vector_db_provider":"qdrant",
"index_name":"my-collection"
}
}

Pricing

The orchestrator charges $0.005 per pipeline run ($5.00 per 1,000 runs). Sub-actors charge separately:

You also pay the embedding provider (OpenAI/Cohere) and vector DB provider (Pinecone/Qdrant) at their standard rates.

Example: Quick Start with Qdrant

Option A — direct text:

{
"text":"Your document content goes here...",
"chunking_strategy":"recursive",
"chunk_size":512,
"embedding_api_key":"sk-...",
"embedding_provider":"openai",
"embedding_model":"text-embedding-3-small",
"vector_db_api_key":"your-qdrant-key",
"vector_db_provider":"qdrant",
"index_name":"my-rag-collection",
"qdrant_url":"https://your-cluster.us-west-1.aws.cloud.qdrant.io:6333"
}

Option B — file upload (Apify Storage, S3, or any public HTTPS URL):

{
"file_url":"https://api.apify.com/v2/key-value-stores/YOUR_STORE_ID/records/document.md",
"chunking_strategy":"markdown",
"chunk_size":512,
"embedding_api_key":"sk-...",
"embedding_provider":"openai",
"embedding_model":"text-embedding-3-small",
"vector_db_api_key":"your-qdrant-key",
"vector_db_provider":"qdrant",
"index_name":"my-rag-collection",
"qdrant_url":"https://your-cluster.us-west-1.aws.cloud.qdrant.io:6333"
}

Sub-Actors

• RAG Content Chunker (source)
• RAG Embedding Generator (source)
• RAG Vector Store Writer (source)

Security

• API keys are validated for presence only and never logged
• Qdrant URLs are validated against cloud.qdrant.io pattern (SSRF prevention)
• All string inputs are sanitized against control characters
• Dataset IDs and field names are validated with strict regex patterns

License

MIT

MCP Integration

This actor works as an MCP tool through Apify's hosted MCP server. No custom server needed.

• Endpoint: https://mcp.apify.com?tools=labrat011/rag-pipeline
• Auth: Authorization: Bearer <APIFY_TOKEN>
• Transport: Streamable HTTP
• Works with: Claude Desktop, Cursor, VS Code, Windsurf, Warp, Gemini CLI

Example MCP config (Claude Desktop / Cursor):

{
"mcpServers":{
"rag-pipeline":{
"url":"https://mcp.apify.com?tools=labrat011/rag-pipeline",
"headers":{
"Authorization":"Bearer <APIFY_TOKEN>"
}
}
}
}

AI agents can use this actor to ingest text into a vector database, build RAG knowledge bases, and set up retrieval-augmented generation pipelines -- all as a single callable MCP tool.