Preventing Google from indexing AJAX HTML fragments/partials.

So you’ve created a neat new site using AJAX. Your server code renders up partial HTML pages which you slap directly into the DOM. Problem is, somehow Google found these partials (even though they are only referenced in Javascript), and has indexed them. Furthermore, because they are not whole HTML pages (and not designed to be), there is no tag where you can place a robots <meta> tag to prevent them being indexed.

Enter the new X-Robots-Tag HTTP header. You can set this header to ‘noindex’ when you return the HTML fragment, so that if google somehow finds it, it still won’t cache it.

Details are here.

I applied this today to geospike.com, Google indexed a few of these <head>-less pages which is pretty terrible if the user clicks on them (they see unstyled crap), hopefully they will drop from the index shortly.


Leave a Reply

Your email address will not be published. Required fields are marked *