Screaming Frog SEO Spider

About

The Screaming Frog SEO Spider is a website crawler for Windows, macOS and Linux, used by thousands of SEOs and agencies for technical SEO site audits. It identifies over 300 SEO issues, warnings and opportunities to improve SEO, website health and user experience. Its capabilities include finding broken links (404s) and server errors, auditing temporary and permanent redirects, analysing page titles and meta descriptions, discovering exact and partial duplicate content, and extracting data from pages using CSS Path, XPath or regex. It also reviews robots.txt and meta robots directives, generates XML and image XML sitemaps, renders JavaScript-heavy sites using an integrated Chromium WRS, visualises site architecture, schedules crawls, and compares crawls and staging against production environments.

You can download the software and crawl up to 500 URLs for free. To remove that limit and access advanced features, you can purchase a licence for £199 per year. Licences last one year and must be renewed afterward. The free version and paid version share the same listed feature set, with the paid version offering unlimited crawling, subject to allocated memory and storage.