Skip to main content
ScrapeGraphAI gives an agent a full web-data toolkit backed by the ScrapeGraphAI API: fetch pages in different formats, pull structured data with an AI prompt, search the web, run multi-page crawls, and schedule change monitors. Use it when your agent needs live web content or LLM-ready extractions without you writing a scraper, browser automation, or search wrapper.

​
Installation

The scrapegraph extra pulls in scrapegraph-py>=2.1.0, which requires Python 3.12 or newer. On Python 3.10 or 3.11, pip install agentor[scrapegraph] succeeds but the SDK is not installed, and instantiating ScrapeGraphAI raises ImportError.

​
Authentication

Set your ScrapeGraphAI API key as an environment variable:
The constructor resolves the key in this order:
  1. The api_key argument, if passed.
  2. The SGAI_API_KEY environment variable (the new SDK default).
  3. The SCRAPEGRAPH_API_KEY environment variable (legacy fallback for pre-2.x deployments).

​
Constructor

​
api_key
str
ScrapeGraphAI API key. Falls back to SGAI_API_KEY, then to SCRAPEGRAPH_API_KEY.

​
Capabilities

Every capability returns a JSON string on success or an Error in <capability>: ... string on failure — so results are always agent-consumable.

​
scrape

Fetch a webpage and return its content.
​
url
str
required
The URL to scrape.
​
format
str
default:"markdown"
Output format: "markdown", "html", "links", or "summary".

​
extract

Extract structured data from a URL using an AI prompt.
​
prompt
str
required
What to extract, e.g. "Extract product names and prices".
​
url
str
required
The page to extract from.
​
schema
dict
Optional JSON schema describing the desired output shape.
Search the web and optionally AI-extract from the results.
​
query
str
required
Search query.
​
num_results
int
default:"3"
Number of results to return (1–20).
​
prompt
str
Optional extraction prompt applied to the results.

​
crawl

Start a crawl job. Returns the crawl id and initial status — poll it with get_crawl_result.
​
url
str
required
Seed URL.
​
max_pages
int
default:"10"
Maximum pages to crawl.
​
max_depth
int
default:"2"
Maximum link depth from the seed.
​
include_patterns
list[str]
Optional path globs to include (e.g. ["/blog/*"]).
​
exclude_patterns
list[str]
Optional path globs to exclude (e.g. ["/admin/*"]).

​
get_crawl_result

Fetch the status and results of a crawl.
​
crawl_id
str
required
Crawl id returned by crawl.

​
stop_crawl / resume_crawl / delete_crawl

Manage a crawl job’s lifecycle by id. Each takes a single crawl_id: str argument.

​
monitor

Create a scheduled monitor that re-scrapes a page on a cron schedule.
​
url
str
required
Page to monitor.
​
interval
str
required
Cron expression, e.g. "0 * * * *" for hourly.
​
name
str
Optional monitor name.
​
webhook_url
str
Optional webhook to receive change notifications.

​
list_monitors / get_monitor / pause_monitor / resume_monitor / delete_monitor

Manage scheduled monitors. list_monitors takes no arguments; the others take a single monitor_id: str.

​
credits

Return plan info and remaining API credits.

​
health

Return API health status.

​
Usage

​
With an Agentor agent

​
Direct capability calls

​
Error handling

Capabilities never raise for API-level failures — the tool converts SDK errors into a string an LLM can act on:
If scrapegraph-py isn’t installed, or if you’re on Python 3.10 / 3.11 where the 2.x SDK cannot be installed, the constructor raises ImportError with an explanation.

​
Source reference

src/agentor/tools/scrapegraphai.py
Last modified on July 17, 2026