Searchine uses self made spider technology to crawl your site and analyse the documents structure and text.
We will take notice of hyperlinks to other pages / documents and follow those as well.
The crawler will take notice of noindex and nofollow requests as it will process robots / XML sitemaps and behave just like any other crawler.
Questions about our crawler and indexing technology
How does Searchine index my site ?
How can I improve the index results ?
Can I prevent some pages from being indexed ?
How often do you crawl my website ?
Do you support robots and sitemap.xml ?
Do you support canonical URL’s ?