• Support
  • FAQ
  •   Login
  •  Corporate site
  • English
  • Nederlands
  •   Login
  •   Corporate site
    English Nederlands
Getting started with Searchine
  • What is Searchine Site Search?
  • Extensible Widgets
  • Supported Content Management Platforms
  • Definitions of terms
Installing
  • Embedded widget
  • Advanced widget
  • Search suggestions widget
  • Examples
  • Content Management Systems plugins
  • Content Security Policy
Crawling & Indexing
  • Crawler setup
  • Add & update content
  • Multiple domains support
  • Crawling frequency
  • Exclude specific URLs
  • Searchine Page Snippet
  • Robots meta tag
  • Add allowed URL parameters
  • Synonyms & stopwords
Frequently asked questions
Crawling & Indexing
Exclude specific URLs
  • Crawler setup
  • Add & update content
  • Multiple domains support
  • Crawling frequency
  • Searchine Page Snippet
  • Robots meta tag
  • Add allowed URL parameters
  • Synonyms & stopwords

Exclude specific URLs

Searchines uses common practices to determine if a page will be indexed or not:

  1. If url is blocked by robots.txt.
  2. If meta robots tag blocks the page from indexing or following.
  3. A custom searchine meta element can be used.
Crawling frequency
Searchine Page Snippet
© Searchine 2025  | 
 Terms of Use
 Privacy Policy
Corporate  |  Contact