Spider Runner

SSE

Configure and run a real multi-page crawl via the FastAPI backend. Streams live log events and scraped items.

Uses Server-Sent Events for real-time streaming. The API backend follows pagination and extracts items using your CSS selectors.

Configuration

Start URL

Item Selector

Next Page Selector

Text Sub-selector

Author Sub-selector

Max Pages (1–20)

Live Log

Logs will appear here when the spider runs…

Equivalent Python

from scrapper_tools.spiders import BaseSpider, Response

class MySpider(BaseSpider):
    name = "my_spider"
    start_urls = ["https://quotes.toscrape.com/"]

    async def parse(self, response: Response):
        for item in response.css(".quote"):
            yield {
                "text": item.css(".text::text").get(),
                "author": item.css(".author::text").get(),
            }
        next_page = response.css(".next a")
        if next_page:
            yield response.follow(next_page[0].attrib["href"])

result = MySpider().run()
result.items.to_json("output.json")