JsonGenius is a robust, self-hosted, AI-powered scraping API written in Go that makes it easy to extract structured data (defined by a JSON Schema) from any webpage. It uses Chromium to render pages like a normal browser, so it works on complex sites.

With JsonGenius and Docker, you can set up a scraping API to pull data from sites in just a few minutes. Define the schema you want, send a URL, and JsonGenius will return extracted data matching your schema. This makes it easy to collect and work with all kinds of web data.

How to use it:

1. Clone JsonGenius from Github and navigating to the jsongenius directory:

git clone https://github.com/semanser/jsongenius
cd jsongenius

2. Insert your OpenAI API Key:

export OPEN_AI_KEY=...

3. Run docker-compose up and the API will be available at http://localhost:3001.

4. To scrape a website, provide its URL and a desired JSON Schema to extract data:

curl -X POST -H "Content-Type: application/json" -d '{
 "url": "/path/to/",
 "schema": {
 "type": "object",
 "properties": {
 "products": {
 "type": "array",
 "items": {
 "type": "object",
 "properties": {
 "name": {
 "type": "string",
 "description": "The product name"
 },
 "price": {
 "type": "number",
 "description": "The price of the product in USD"
 }
 }
 }
 }
 }
 }
}' http://localhost:3001/lookup

More Like This