592000 Crawl AJAX Pages and All Links of Your Sitemap with Site Audit

Start Here

Find the Right Tools

Platform

Top Apps

Grow with Semrush

Explore Free Tools

About Semrush

Main page /Product News

Crawl AJAX Pages and All Links of Your Sitemap with Site Audit

August 2, 2017
Crawl AJAX Pages and All Links of Your Sitemap with Site Audit
We are proud to announce that we’ve implemented two new crawling features in the Site Audit tool: AJAX crawling and Sitemap links crawling.

You can now crawl your entire website without missing a piece of dynamically generated content or a URL, and immediately take action on technical issues.

This is a huge step in our tool development, since only a few tools on the market provide this functionality and now the SEMrush Site Audit tool is one of them.

Let’s get straight to the details.

Crawl your AJAX pages

AJAX crawling has always been a struggle for SEO specialists as search engines didn’t understand the pages that use JavaScript to show content to users until 2009. This is when Google offered its AJAX crawling scheme and thus put extra duty on SEOs to ensure their JS-content is indexed by search engines.

Fortunately since 2015 Google started to render and understand the pages with dynamic content like modern browsers and eventually deprecated the old AJAX crawling scheme.

However, it’s still crucial for SEO professionals to conduct the full-scale analysis to check if Google sees these pages correctly. Now with the new AJAX crawling feature in Site Audit, you can track your dynamic content and your internal linking health without extra efforts!

S1.png
If you don’t have a project with Site Audit setup, head to the Projects dashboard, create a new project and set up the Site Audit tool. From now on, your AJAX pages will be crawled automatically.

Crawl all links of your sitemap

With this update, you can now send our crawler directly to your sitemap file to make it crawl your website exactly in accordance with your sitemap the same way as you recommend Google to do it.

In order to crawl all links of your sitemap, first go to your project with Site Audit setup, or create a new one. Click on the gear wheel and select ‘Crawl sources’.
S2.png
Now you can change your crawling settings — click on ‘Crawl source’ and select:

‘Sitemaps on site’ to make SEMrush find it automatically

‘Enter sitemap URL’ to enter the URL manually
S3.png
Finally, click on the ‘Save’ button and re-run your campaign. Your website is now crawled in accordance with your sitemap.

Here at SEMrush, we are sure that this new functionality will help you conduct the most complete SEO analysis of your website and not a single page will be left unnoticed by our crawlers. And this is one more step in making the Site Audit tool the best site crawler on market.

What do you think about this update? Share your thoughts and ideas at
site-audit-feedback@semrush.downloader.la, and stay tuned!
Crawl your entire website
Set up Site Audit

About Semrush

Semrush is a leading online visibility management SaaS platform that enables businesses globally to run search engine optimization, pay-per-click, content, social media and competitive research campaigns and get measurable results from online marketing. Semrush offers insights and solutions for companies to build, manage, and measure campaigns across various marketing channels. Semrush, with over 94,000 paying customers, is headquartered in Boston and has offices in Philadelphia, Trevose, Austin, Dallas, Amsterdam, Barcelona, Belgrade, Berlin, Limassol, Prague, Warsaw, and Yerevan.

Relevant news

Semrush Launches MCP Connector in Perplexity, Integrating Search Intelligence Within the AI Search Engine

The new Model Context Protocol (MCP) connector enables all Perplexity users in Computer to access Semrush’s search intelligence data natively inside the LLM platform to boost brand visibility. 
Product News
June 3, 2026

Semrush for Enterprise Unifies Content Optimization Capabilities Across SEO and AI Search

New workflows shared between Enterprise SEO and AI Optimization enable teams to grow brand visibility through high-performing content for both search channels.
Product News
May 27, 2026

Semrush Expands AI Visibility Database To 32 Countries, With 17 New Regional Markets

Global coverage expansion now provides international teams with real AI search data for previously unmeasured markets, growing the Semrush prompt database to more than 261 million LLM prompts.
Product News
May 14, 2026