Robots meta tag
Searchine supports the robots meta tag.
The meta robots tag can be used to inform crawlers if the page should be indexed or not and what links to follow.
If you do not place a meta robots tag on your website, it means robots will index every page and follow all links on the page's they can find.
The following commands can be combined.
- FOLLOW – a command for the search engine crawler to follow the links in that webpage
- INDEX – a command for the search engine crawler to index that webpage
- NOFOLLOW – a command for the search engine crawler NOT to follow the links in that webpage
- NOINDEX – a command for the search engine crawler NOT to index that webpage
Example:
<meta name="robots" content="index, follow" />
We also have our own meta tag for specifying specific behaviour for the Searchine bot.
You should use this in cases you want other public search services to be able to index more or less content on your site then you want in your internal search results.
<meta name="robots" content="index, follow" />
<meta name="searchinebot" content="noindex, follow" />
Or the other way around:
<meta name="robots" content="noindex, nofollow" />
<meta name="searchinebot" content="index, follow" />