Multi-Agent Web Scraper
Three agents, three jobs: one finds the pages, one copies out the data, one checks and cleans it. Pick a sample website and watch them work.
New here? Read the short guide for this demo, with steps to follow and what to look for.
The idea in one minute
Think of a small research team.
One person goes through a website page by page. A second copies the useful details into a sheet. A third checks the sheet: tidies the prices, spots duplicates and throws out rows with something important missing.
A multi-agent scraper splits the work the same way. Each agent has one clear job, so each job is easy to check, and when something goes wrong you can see exactly where.
- 1Pick a sample website and look around it.
- 2Run the crew: Navigator, Extractor, Validator.
- 3Break the page and watch the checks catch it.
This is a demo with made-up data, and it runs only in your browser. The three websites, their books, events, jobs, names and numbers are invented, and nothing is fetched from the internet. The crawling, extraction and cleaning are real, though: they run live on the sample pages, including any changes you make. No AI model is called, and nothing you type is sent anywhere or stored.
1 Pick a website
View the page's HTML
2 Run the crew
Navigator follows the listing pages and respects robots.txt
Extractor copies each field exactly as written
Extraction rules (CSS selectors)
Validator cleans, removes duplicates, rejects incomplete rows
Stopped and asked a person to check. The page layout looks different from what the rules expect, so the data would come out half empty:
A real crew would alert someone to update the rules, or let an AI model suggest new ones for a person to approve.
What was cleaned
What was left out, and why
3 Break the page
Websites change without warning. Try a change to page 1 with a button, or edit its HTML yourself, then run the crew again.
Scraping responsibly
Polite, legal and careful.
- Read the website's terms and its robots.txt first, and do what they say.
- Go slowly: a pause between pages, and a limit on how many pages are opened.
- Collect only what you need. Leave personal details such as names and phone numbers alone unless you have a lawful reason to keep them.
- Prefer an official API or a data feed when the website offers one.
Honest limits
What this demo does not do.
- Nothing is fetched over the internet. A real crawler also waits between requests, retries failures and handles logins.
- Many real websites build their pages with JavaScript, which needs a headless browser rather than reading the HTML.
- Here each agent follows plain rules. In a real build, an AI model can help find fields when a layout changes, but the checks still decide what gets saved.
Have an idea? Let's build it properly.
Tell us what you want to build or automate. We'll reply with practical next steps.