Freshness matters. It matters to searchers, therefore it matters to search engines, therefore it should matter to you. The best tool to ensure that search engines have your site’s freshest content is your sitemap. This post explains why pinging search engines with your sitemaps is important for SEO and how to optimize the process and stay fresh.
Why Sitemaps? Why Ping them?
You have probably already submitted your sitemap to Google, Bing and other search engines using their webmaster consoles, if so, you’re in a good spot— rest assured that search engines know that you have a sitemap. And if you have a site that is relatively static, meaning that pages are not being updated or created often, this is probably good enough. Depending on the perceived value of your site from a search engine’s perspective, search engines are probably requesting your sitemap(s) at a minimum, close to once a day, or potentially several times a day.
If you have a site that changes frequently then pinging search engines with your sitemaps is important. It is in your, and Google’s, best interest that they have the newest and most up-to-date content in their index, available to serve in search results. Examples of this would be marketplace sites where new products or services are being listed all the time, currency exchange listing sites where pages are continuously updating, or news publishers where freshness is critical.
To ensure that Google and other search engines know when your site has new content pinging them with your recently-updated sitemap is a must.
How pinging works
It is actually very simple. If you have ever used a web browser you will probably understand.
- You send an HTTP GET request to a specified “ping” URL on a search engine’s server with your sitemap’s URL appended to it.
- When the search engine receives the request, they, in turn, send an HTTP GET request to the sitemap URL that you submitted.
- If your sitemap returns a 200 “OK” response, with a valid sitemap, the search engine will scan the URL’s on the sitemap and reconcile them against the URLs that they have previously discovered.
- The search engine may decide to crawl some, all, or none of the of URL’s listed in the sitemap.
It is important to note that pinging a sitemap does not guarantee that the sitemap URL’s will be crawled. Google says, “Please note that we do not add all submitted URLs to our index, and we cannot make any predictions or guarantees about when or if they will appear.” It is pretty safe to say that all the URLs will be recognized though. That being said, crawl rate has a tendency to increase sharply after you notify search engines that they are fresh.
How often to ping
Google used to recommend pinging no more than once per hour in their sitemaps documentation. As of 2015, they do not suggest any rate limit. Things have changed…
At Postmates, while testing Airflow data pipelines, we pinged our sitemaps 174 times in one day. Looking at our server logs, we recognized that, every single attempt to ping the sitemaps lead to a subsequent GET request from Googlebot. This was true for each of our sitemaps— every time.
The subtle nuance though, is that, since our sitemaps did not actually change during that period – and had been cached – the server returned a 304 (Not Modified) response and did not actually send a sitemap. This happens because Googlebot sends a request with a Last-Modified header as a comparison for the freshness of the sitemap file. The value of the Last-Modified header is the same as the last time that you pinged Google. This is Google saying essentially, “as long as the sitemap file has changed since I looked last time, I’m not interested.”
Stay Fresh
The takeaway is that if you really want to keep Google’s index of your site fresh, ping Google with your sitemaps. Do it as often as you have fresh content and make sure that the content listed in the sitemap file is discernibly different than last time you pinged Google. In addition, make sure that you set the appropriate headers with the updated sitemap to ensure that Googlebot will actually receive the new sitemap.
If you are ready to start pinging search engines, here are the URLs to do it:
https://www.google.com/webmasters/sitemaps/ping?sitemap=<sitemap_url>
https://www.bing.com/ping?sitemap=<sitemap_url>
If you want to tell Google about URLs manually, you can click these links and then append the full URL of your sitemap or sitemap index files.