Crawler documentation

Know which Gemmetric crawler is visiting, what it can access, and why.

Gemmetric uses one crawler name for public website checks and another for sites a customer has verified or is allowed to manage. Neither crawler can access private pages.

You decide whether to allow the traffic. Your robots.txt, rate limits, and firewall rules remain in control.

Clearly named crawlersPublic pages onlyYour rules stay in control
Public pages onlyTwo crawler names
Public website checks

GemmetricPublicScanner

Free Snapshots and Prospect Audits

GemmetricPublicScanner (+https://gemmetric.ai/crawler)
Sites you manage

GemmetricBot

Checks for verified or authorized sites

GemmetricBot (+https://gemmetric.ai)
GET requestsrobots.txt respectedNo private content

Two crawler names

The User-Agent shows why Gemmetric visited the site.

GemmetricPublicScanner checks public websites. GemmetricBot checks sites that a customer has verified or is authorized to manage. GemmetricBot also signs those requests so supported security services can verify them.

GemmetricPublicScanner

Used for Free Snapshots and Prospect Audits that check information already available on the public web.

GemmetricBot

Used for scans on sites that the customer has verified or is authorized to manage.

Signed GemmetricBot requests

Web Bot Auth lets supported security services verify a managed-site request instead of trusting the User-Agent name alone.

What the crawlers access

Both crawlers request only public website pages and files.

They use those pages and files to see whether a website can be reached and understood. They do not sign in or request account-only content.

Public pages

Gemmetric requests pages that anyone can visit without signing in.

robots.txt and sitemaps

The crawlers read the site's published robots.txt and sitemap files and follow the rules that apply to them.

GET requests

The crawlers use GET requests to retrieve public pages and files. They do not submit forms or make changes to the site.

No private content

Gemmetric does not sign in to the site or request private, authenticated, or account-only pages.

If a request is blocked

Check the crawler name and your site settings.

A blocked request can simply mean your current rules do not allow it. You can keep it blocked or change the relevant setting if you want Gemmetric to scan those public pages.
  1. 01

    Identify the crawler

    Check the User-Agent to see whether the request came from GemmetricPublicScanner or GemmetricBot.

    Know which crawler visited
  2. 02

    Find out why it was blocked

    Check robots.txt, rate limits, firewall rules, and any security challenge affecting the public page.

    Find the rule or setting
  3. 03

    Decide what to allow

    Keep the request blocked, change the relevant rule, or contact Gemmetric when the traffic does not match what you expect.

    Keep control of your site

Crawler boundaries

Here is what Gemmetric's crawlers are not used for.

The crawler name should tell a site owner or security team what the traffic is for. It should not require guesswork.

No AI training

Gemmetric uses the pages for its website checks, not to train an AI model.

No bulk scraping

The crawlers are not used to build a general search index, resell content, or collect it for unrelated purposes.

No bypass promise

Public checks may be blocked by robots.txt, rate limits, Cloudflare, or another WAF. Gemmetric does not promise to bypass those controls.

Crawler questions

Use the crawler name and your own site rules to decide what to allow.

These answers cover the most common questions. Contact support if a request does not match what is documented here.
What is GemmetricPublicScanner?

GemmetricPublicScanner checks public pages for Free Snapshots and Prospect Audits. A site owner can block it through robots.txt, rate limits, Cloudflare, or other firewall rules.

What is GemmetricBot?

GemmetricBot scans sites that a customer has verified or is authorized to manage. It still requests only the public pages and files needed for the website check.

Can I verify that a GemmetricBot request is genuine?

GemmetricBot requests for sites a customer has verified or is authorized to manage use Web Bot Auth signatures. Supported security services can verify the signature through the public key directory advertised in the request. A User-Agent by itself should not be treated as proof because it can be copied.

Is GemmetricBot recognized by Cloudflare?

Yes. GemmetricBot is listed in Cloudflare's Verified Bot Directory. The public directory entry gives site owners and supported security services an independent way to recognize Gemmetric's crawler identity.

Does Gemmetric access private or logged-in pages?

No. Gemmetric's crawlers do not log in to the scanned website or collect private, authenticated, or account-only content.

Does Gemmetric respect robots.txt?

Yes. Gemmetric retrieves robots.txt and applies the relevant published rules to the crawler identity being used.

What should I do if a Gemmetric crawler is blocked?

First confirm the User-Agent, then review robots.txt, rate limits, CDN, firewall, and challenge settings. Site owners may keep the traffic blocked. If you or your client asked Gemmetric to scan the site and the behavior is unclear, contact support@gemmetric.ai.

Need help reviewing Gemmetric crawler traffic?

Send us the affected URL, timestamp, User-Agent, and response details. Do not send passwords, private page content, or access credentials.