ScrapeGraphAI gives an agent a full web-data toolkit backed by the ScrapeGraphAI API: fetch pages in different formats, pull structured data with an AI prompt, search the web, run multi-page crawls, and schedule change monitors.
Use it when your agent needs live web content or LLM-ready extractions without you writing a scraper, browser automation, or search wrapper.
Installation
The
scrapegraph extra pulls in scrapegraph-py>=2.1.0, which requires Python 3.12 or newer. On Python 3.11, pip install --pre "agentor[scrapegraph]" succeeds but the SDK is not installed, and instantiating ScrapeGraphAI raises ImportError.Authentication
Set your ScrapeGraphAI API key as an environment variable:- The
api_keyargument, if passed. - The
SGAI_API_KEYenvironment variable (the new SDK default). - The
SCRAPEGRAPH_API_KEYenvironment variable (legacy fallback for pre-2.x deployments).
Constructor
str
ScrapeGraphAI API key. Falls back to
SGAI_API_KEY, then to SCRAPEGRAPH_API_KEY.Capabilities
Every capability returns a JSON string on success or anError in <capability>: ... string on failure — so results are always agent-consumable.
scrape
Fetch a webpage and return its content.str
required
The URL to scrape.
str
default:"markdown"
Output format:
"markdown", "html", "links", or "summary".extract
Extract structured data from a URL using an AI prompt.str
required
What to extract, e.g.
"Extract product names and prices".str
required
The page to extract from.
dict
Optional JSON schema describing the desired output shape.
search
Search the web and optionally AI-extract from the results.str
required
Search query.
int
default:"3"
Number of results to return (1–20).
str
Optional extraction prompt applied to the results.
crawl
Start a crawl job. Returns the crawl id and initial status — poll it withget_crawl_result.
str
required
Seed URL.
int
default:"10"
Maximum pages to crawl.
int
default:"2"
Maximum link depth from the seed.
list[str]
Optional path globs to include (e.g.
["/blog/*"]).list[str]
Optional path globs to exclude (e.g.
["/admin/*"]).get_crawl_result
Fetch the status and results of a crawl.str
required
Crawl id returned by
crawl.stop_crawl / resume_crawl / delete_crawl
Manage a crawl job’s lifecycle by id. Each takes a singlecrawl_id: str argument.
monitor
Create a scheduled monitor that re-scrapes a page on a cron schedule.str
required
Page to monitor.
str
required
Cron expression, e.g.
"0 * * * *" for hourly.str
Optional monitor name.
str
Optional webhook to receive change notifications.
list_monitors / get_monitor / pause_monitor / resume_monitor / delete_monitor
Manage scheduled monitors.list_monitors takes no arguments; the others take a single monitor_id: str.
credits
Return plan info and remaining API credits.health
Return API health status.Usage
With an Agentor agent
Direct capability calls
Error handling
Capabilities never raise for API-level failures — the tool converts SDK errors into a string an LLM can act on:scrapegraph-py isn’t installed, or if you are on Python 3.11 where the 2.x SDK cannot be installed, the constructor raises ImportError with an explanation.
Source reference
src/agentor/tools/scrapegraphai.py