Free · Open source
Technical SEO crawler
Crawl your site the way a search engine reads it, and get the technical issues that hold back visibility in one report. Runs on your own machine, so nothing leaves it.
Python 3.11 · MIT licensed
Why crawl your own site
Search engines have to reach each page, read it, and decide whether to index it. Most of that happens out of sight. When something in the plumbing breaks, a redirect that loops, a page quietly set to noindex, a canonical pointing at the wrong URL, pages drop out of results and nobody sends you a warning.
A crawl shows you the site the way the crawler sees it, before the gaps cost you traffic. You run it, you read the report, and you fix the handful of things that matter.
What it checks
Indexability and crawling
Status codes, redirects, redirect loops, canonicals, noindex, and orphan pages. Which pages a search engine can actually reach and keep.
On-page tags
Titles, descriptions, and headings sized in pixels, plus hreflang and Open Graph. The tags that decide how you show up.
Duplicate content
Groups near-identical pages by title, heading, and body, so you can spot pages competing with each other.
Performance
Core Web Vitals per page, with Lighthouse and real-user data when you add a free PageSpeed key.
Structured data and tracking
JSON-LD, security headers, accessibility, and analytics tags for GA4, Tag Manager, Search Console, and Bing.
JavaScript rendering
Loads each page in a real browser and shows what only appears after JavaScript runs.
How to run it
It's a small Python program you run from the command line. Install it once, then point it at a site in one of three ways.
List
Check a specific set of pages from a file of URLs.
Spider
Follow links from a starting page and map the whole site.
Sitemap
Audit every URL you already publish in your sitemap.
# install
pip install -r requirements.txt
# spider a whole site into an Excel report
python crawler.py --mode spider \
--seed https://example.com/ \
--output audit.xlsxYou get an Excel workbook with one row per URL, plus HTML and Markdown reports that summarise and prioritise the findings. Full setup and options are in the repository README.
What you'd use it to investigate
A crawl is most useful when you have a question. A few common ones.
Traffic fell after a site migration
Check the redirects for chains and loops, and confirm the new URLs canonicalise to themselves rather than the old ones.
Pages aren't getting indexed
Look for noindex directives, canonicals pointing elsewhere, and orphan pages with no internal links leading to them.
Two pages compete for the same term
Use the duplicate-content grouping to find pages with matching titles, headings, or body text that split their ranking signals.
The site feels slow
Read Core Web Vitals and Lighthouse scores per URL to see which pages, and which metrics, are dragging.
An international site
Check hreflang coverage across locales and catch the mismatches that send the wrong language version to search engines.
A Shopify storefront
Crawl the storefront with authenticated access to audit product pages, structured data, and duplicate variants at scale.
The scores are local approximations. They point you at what to look at. They don't promise a ranking or a sale. And only crawl sites you own or have permission to test.
Want these findings read against your paid and organic search and turned into a plan? Addy reasons through the data, proposes the next steps, and acts with your approval.