Pricing

Rating

Developer

Actor stats

Categories

Share

Congress Trading Pipeline — API

A senator buys $250k in defense stock the week before a major procurement vote. The filing lands quietly on the Senate EFD system.

This pipeline catches it — normalized, deduplicated, queryable JSON — within hours of the official disclosure. Public domain government data, no middleman.

Who uses this

• Traders following Senate insiders — committee members moving before contract announcements, votes, and regulatory decisions
• Algo traders who want structured JSON they can pipe directly into a strategy without manual CSV parsing
• Data engineers who need a clean, deduplicated Senate trading feed with stable record IDs for joins and incremental loads
• App developers who want a drop-in REST API — run on Railway or Render, point your frontend at /api/transactions

Why this instead of existing tools? Senate EFD data is public but awkward to consume. This pipeline normalizes raw filings into a consistent schema with stable IDs, dedup, and a queryable REST API — so you build on top, not around.

Prerequisites

• Node.js 18+
• No external services, databases, or API keys required for MVP

Setup

npminstall
cp .env.example .env # edit as needed — all vars have defaults
npm run dev

Server starts on http://localhost:3001.
On first boot the scheduler runs the pipeline immediately, then every 6 hours.

Environment variables

Pipeline architecture

┌──────────┐ ┌──────────┐ ┌─────────────┐ ┌────────┐ ┌────────┐
│ Fetch │──▶│ Parse │──▶│ Transform │──▶│ Dedup │──▶│ Store │
│ │ │ │ │ │ │ │ │ │
│ Senate │ │ JSON │ │ type│ │ key: │ │ SQLite │
│ EFDAPI │ │ primary │ │ amount │ │ name + │ │ INSERT │
│ GET │ │ │ │ dates │ │ date + │ │ OR │
│ 100/page │ │ HTML │ │ owner │ │ asset +│ │ IGNORE │
│ │ │ fallback │ │ ticker │ │ amount │ │ │
└──────────┘ └──────────┘ └─────────────┘ └────────┘ └────────┘
 │
 ▼
 ┌──────────────┐
 │ Express API │
 │ :3001 │
 └──────────────┘

Source endpoint: GET https://efts.senate.gov/LATEST/search-index
Pagination: 100 records/page, loops until hits.total exhausted
Fallback: if JSON parse yields empty asset_name on all rows, re-parses raw HTML
Retry: 3 attempts with exponential backoff + ±25% jitter on all HTTP calls

API reference

GET /health

$curl http://localhost:3001/health

{
"status":"ok",
"db_count":847,
"last_run":"2026-04-29T14:23:00.000Z"
}

GET /api/refresh

Returns timestamp of most recently stored record. Called by the frontend on every page mount.

$curl http://localhost:3001/api/refresh

{"lastUpdated":"2026-04-29T14:23:00.000Z"}

lastUpdated is null if no records exist yet.

POST /api/refresh

Triggers a full pipeline run. Called when the user clicks "Refresh Data" in the frontend.

$curl-X POST http://localhost:3001/api/refresh

{"ok":true,"signals":14,"lastUpdated":"2026-04-29T14:23:00.000Z"}

On failure:

{"ok":false,"error":"Fetch failed: HTTP 503 Service Unavailable"}

GET /api/cron

Same pipeline run as POST /api/refresh, protected by CRON_SECRET. Called by an external scheduler (Cloudflare Worker, cron job, etc.).

curl-H"x-cron-secret: your-secret" http://localhost:3001/api/cron
# or
curl"http://localhost:3001/api/cron?secret=your-secret"

{
"ok":true,
"summary":{
"ingested":340,
"newTrades":14,
"signalsGenerated":14,
"topScore":null,
"topScoreTicker":null,
"runAt":"2026-04-29T14:23:00.000Z"
}
}

Returns 401 if secret is missing or wrong.

GET /api/sync-committees

Syncs congressional committee membership. Protected by CRON_SECRET. Run once on setup, then weekly.

$curl-H"x-cron-secret: your-secret" http://localhost:3001/api/sync-committees

{"ok":true,"synced":0}

GET /api/transactions

Queryable read endpoint. Returns transactions serialized to match the frontend Signal field names.

# All recent transactions (default limit 500)
curl http://localhost:3001/api/transactions
# Filter by ticker
curl"http://localhost:3001/api/transactions?ticker=AAPL"
# Filter by politician (LIKE match, case-insensitive)
curl"http://localhost:3001/api/transactions?politician=Pelosi"
# Date range
curl"http://localhost:3001/api/transactions?date_from=2026-04-01&date_to=2026-04-30"
# Type + owner + pagination
curl"http://localhost:3001/api/transactions?type=buy&owner=joint&limit=50&offset=0"

{
"count":2,
"data":[
{
"id":"a3f...c1",
"filer_name":"Nancy Pelosi",
"filer_type":"congress",
"trade_type":"purchase",
"ticker":"NVDA",
"asset_name":"NVIDIA Corporation",
"asset_type":"Stock",
"amount_low":1000001,
"amount_high":5000000,
"amount_midpoint":3000000,
"trade_date":"2026-04-29",
"filing_date":"2026-04-29",
"owner":"joint",
"is_active":true
}
]
}

Query parameters:

Invalid params return 400:

{"error":{"date_from":["Must be YYYY-MM-DD"]}}

GET /api/debug

Dev diagnostics. No auth. Returns DB count and 2 sample records.

$curl http://localhost:3001/api/debug

Cron schedule

Default: 0 */6 * * * (every 6 hours).

Change via CRON_SCHEDULE env var — any valid node-cron expression.

CRON_SCHEDULE="0 */2 * * *"npm run dev # every 2 hours
CRON_SCHEDULE="0 8 * * *"npm run dev # once daily at 08:00

Last run stats (timestamp, inserted, skipped, errors) are persisted to ./data/last_run.json after each run.

Seeding and smoke test

Load 20 realistic fake records covering edge cases (null tickers, spouse/child owners, large amounts, same-day multi-trades, clusters):

$npm run seed

Verify the running server responds correctly:

# Terminal 1
npm run dev
# Terminal 2
npm run smoke

Smoke test exits 0 on all pass, 1 on any failure.

Phase 2 roadmap

House of Representatives disclosures (efd.house.gov) use a different filing format and will be added after Senate coverage is stable. Planned additions: PDF parsing for older PTRs that lack structured data, ticker enrichment via OpenFIGI or a static CUSIP mapping table (resolving the ticker: null cases currently stored as-is), a scoring engine that ranks transactions by conviction signal (cluster detection, filing delay, filer track record), and Telegram/email alerts for high-score transactions. Multi-tenant auth (Supabase RLS + Paddle billing) is tracked separately under the SaaS roadmap.

Data source

All data is sourced from the U.S. Senate Electronic Financial Disclosures system — a public government database. Senate PTR filings are required under the STOCK Act and are public domain. This pipeline does not scrape third-party aggregators.