Crawling & Indexing

Setting up the crawler

Getting started with the Searchine crawler.

Read more

Exclude specific URLs

Read more

Searchine Page Snippet

The page snippet is used to pass extra information to Searchine regarding the crawled page

Read more

Robots meta tag

Searchine supports the robots meta tag.

Read more