Screaming Frog Log Analyzer

Overview

Screaming Frog Log File Analyzer is a companion product to the popular Screaming Frog SEO Spider. While the Spider crawls websites like a search engine would, the Log Analyzer examines how search engines actually crawled your site by parsing server access log files. This real-world crawl data reveals insights that no synthetic crawl can replicate.

Understanding how Googlebot and other search engine bots interact with your site is crucial for technical SEO, particularly for large sites where crawl budget management directly impacts indexation and ranking. The Log Analyzer makes this data accessible without requiring command-line log processing skills.

Key Features

The Log Analyzer imports server log files in Apache, Nginx, IIS, and custom formats. It parses millions of log entries and filters them by bot type, showing exactly which URLs Googlebot, Bingbot, and other crawlers requested, when they visited, and what response codes they received.

The crawl frequency analysis shows how often each URL is crawled, identifying pages that receive too much or too little bot attention. This is essential for crawl budget optimization — if Googlebot spends most of its time on low-value pages (faceted navigation, parameter URLs, old blog posts), high-value pages may not be crawled frequently enough.

Response code analysis surfaces 404 errors, redirect chains, and server errors that bots encounter. These issues are invisible in standard analytics because they only occur during bot crawls, not user visits.

The integration with Screaming Frog SEO Spider is where the Log Analyzer becomes exceptionally powerful. By combining spider crawl data with log file data, users can identify orphan pages (URLs crawled by bots but not linked from the site), discover content that search engines cannot find (linked pages never crawled), and compare what the site exposes versus what bots actually access.

Visualization features include crawl trend charts, directory-level crawl distribution, and status code breakdowns. Reports can be exported for stakeholder presentations.

Pricing

The free version analyzes up to 1,000 log events. The paid license costs $139 per year and removes all limits, allowing analysis of log files with millions of entries. The license is per machine, matching the Screaming Frog SEO Spider licensing model.

Ideal Use Cases

The Log Analyzer is essential for technical SEOs managing large websites — e-commerce sites with millions of product pages, news publications with extensive archives, and enterprise sites with complex URL structures. It is particularly valuable during site migrations, where verifying that bots can find and crawl new URLs is critical. Crawl budget audits rely heavily on log file data that only this type of tool can provide.

Limitations

Like its sibling product, the Log Analyzer is a desktop application without cloud or collaboration features. Results cannot be easily shared with team members who do not have the software installed. The tool requires access to raw server log files, which can be difficult to obtain from some hosting providers or CDN configurations. The visualization capabilities are functional but not as polished as cloud-based alternatives like Botify or OnCrawl that offer richer dashboards and automated analysis.