592000 Site Audit: URL Settings and Crawl Delay

Start Here

Find the Right Tools

Platform

Top Apps

Grow with Semrush

Explore Free Tools

About Semrush

Main page /Product News

Site Audit: URL Settings and Crawl Delay

December 11, 2016
Site Audit: URL Settings and Crawl Delay
The latest update to our Site Audit tool introduces two much-demanded features that allow users to more finely adjust crawler settings: advanced URL parameter exclusion settings and the crawl delay feature.

Advanced URL Parameters Exclusion 

You can now specify what URL parameters you want the Site Audit crawler to ignore. For user convenience, the new parameter removal feature works in a similar way as the URL parameter settings in the Google Search Console. So, for instance, if you set up Site Audit to ignore all UTM codes, the crawler will process http://example.com/webpage.html and http://example.com/webpage.html?utm_media=blog as the same URL.
You can create or edit your list of excluded URL parameters by clicking on the gear icon in the Site Audit interface and then selecting the 'Removed parameters' option in the dropdown menu.
Then, you can specify URL parameters you want removed from URLs before they are crawled by typing in one parameter per line. When you're done, click the 'Save' button and re-run your Site Audit to apply all changes.

Crawl Delay

By default, the Site Audit crawler ignores the 'Crawl Delay' parameter specified in a robots.txt file in order to speed up the crawling process. However, in rare cases such crawling intensity might cause a website to overload, so in order to prevent that, users can now use the 'Respect robots.txt crawl delay' rule in the Site Audit settings.
You can do so by clicking on the gear icon in the Site Audit interface and selecting the 'Respect robots.txt crawl delay' option in the dropdown menu.
This will open the Site Audit settings. From there, tick the 'Respect robots.txt crawl delay' option and click on the 'Save' button to apply new settings.
We hope these new features improve your Site Audit experience! Please feel free to send your comments, suggestions and questions to site-audit-feedback@semrush.downloader.la Thank you for helping us make SEMrush better!

About Semrush

Semrush is a leading online visibility management SaaS platform that enables businesses globally to run search engine optimization, pay-per-click, content, social media and competitive research campaigns and get measurable results from online marketing. Semrush offers insights and solutions for companies to build, manage, and measure campaigns across various marketing channels. Semrush, with over 91,000 paying customers, is headquartered in Boston and has offices in Philadelphia, Trevose, Austin, Dallas, Amsterdam, Barcelona, Belgrade, Berlin, Limassol, Prague, Warsaw, and Yerevan.

Relevant news

Semrush Launches MCP Connector in Perplexity, Integrating Search Intelligence Within the AI Search Engine

The new Model Context Protocol (MCP) connector enables all Perplexity users in Computer to access Semrush’s search intelligence data natively inside the LLM platform to boost brand visibility. 
Product News
June 3, 2026

Semrush for Enterprise Unifies Content Optimization Capabilities Across SEO and AI Search

New workflows shared between Enterprise SEO and AI Optimization enable teams to grow brand visibility through high-performing content for both search channels.
Product News
May 27, 2026

Semrush Expands AI Visibility Database To 32 Countries, With 17 New Regional Markets

Global coverage expansion now provides international teams with real AI search data for previously unmeasured markets, growing the Semrush prompt database to more than 261 million LLM prompts.
Product News
May 14, 2026