Web Spiders and Dynamic Content

Since I have seperated areas of this website into specific areas I have started to think about how the pages are indexed by Search Engines such as Google. While I am not particularly worried that my pages are regularly "spidered" and they can be found by Internet users making a search I am mindful that they are found and I possibly need to modify my templates accordingly.

Update: March 2021

I have observered that I don't really have to worry as my pages seem to be "spidered" quite adequately. This could be due to my increased use of Google Search Console and the addition of a more comprehensive sitemap scheme.

Update: September 2023

Revisiting this topic when I was checking and validting my pages I see that this question comes up when I make a search for "web spiders and dynamic content" I see my page in the Google SERP.

In addition I see that screamingfrog.co.uk do actually address this problem and state that it used to be the case that this was a problem with Javascript generated content.

There a few technologies to investigate here. Also the statement about rendering pages like a modern-day browser before indexing.

My original thoughts

Header Text and Header Paragraphs

The question is how a Web Spider reads the content of a page - Javascripts may or maynot be executed. If I have active content it might not be indexed appropriatly.

When a search is made on Google the search terms are checked against the index that Google generates for the whole web. It generates this index by spidering web pages on a routine basis. The exact process that Google use to build the index is known only to Google but we do know as webmasters that the method is constantly changing and the famous Google Algorithm is applied to the content that it collects from webdites. There are various theories and advice that can be found on-line relating to SEO (Search Engine Optimisation).

Google also publish Webmaster Guidelines and their stated reasons for their approach is to stop webmasters "gaming" the system and writing code to influence the way that their pages are found in searches. This is a debate in itself, I only say that Google are business and they are likey to do things that are in their interest and their methods are not altruistic.

Sample Searches

To analyse my point I am thinking of a few examples. I recently changed the header text on my web design pages from BW Web Design to Tempusfugit Web Design - this was due to the fact that there was an East London web design company with that name.

The page Title and H1 tags are the subject of tue page and it is the header text that is replaced with custom text dependant on the context of the page. The text read by the web spider should give the indexing engine enough information to allow the page to be found by a search. This is also in keeping with the Google Webmaster Guidelines and there are no underhand techniques such as keyword stuffing.

The likelyhood of one of my pages on web design being found by someone making an internet search is remote as there are so many other pages on the internet hosted by sites with far greater authority than mine. A page with a unique subject and content may well be found. I know this as pages with Belchamp Walter featuring on them are found.

Another example that may be found is one featuring a specific error that occurs when the antenna of a DJI Mavic Pro drones controller becomes magnetized causing a STICK EMI 1 error.

My Love-Hate relationship to Social Media

I have made my views on this clear in the past. Google and its part on what people see from the Internet when they make a search are included in this.

In the past I may have attempted to get my pages "rated" by Google.