You might remember the plethora of different search engines which floated around the internet in the mid-nineties.
These search engines were pretty primitive by today’s standards. Since the web was a much smaller place back then, many ‘search engines’ were just Yellow Pages-style directories which were compiled by humans. Crawlers (aka spiders) arrived in 1993 – these would index the content of your site and follow any links you had to other sites so it could index them too. Continue Reading