GemmetricPublicScanner
Free Snapshots and Prospect Audits
GemmetricPublicScanner (+https://gemmetric.ai/crawler)Crawler documentation
Gemmetric uses one crawler name for public website checks and another for sites a customer has verified or is allowed to manage. Neither crawler can access private pages.
You decide whether to allow the traffic. Your robots.txt, rate limits, and firewall rules remain in control.
Free Snapshots and Prospect Audits
GemmetricPublicScanner (+https://gemmetric.ai/crawler)Checks for verified or authorized sites
GemmetricBot (+https://gemmetric.ai)Two crawler names
Used for Free Snapshots and Prospect Audits that check information already available on the public web.
Used for scans on sites that the customer has verified or is authorized to manage.
Web Bot Auth lets supported security services verify a managed-site request instead of trusting the User-Agent name alone.
What the crawlers access
Gemmetric requests pages that anyone can visit without signing in.
The crawlers read the site's published robots.txt and sitemap files and follow the rules that apply to them.
The crawlers use GET requests to retrieve public pages and files. They do not submit forms or make changes to the site.
Gemmetric does not sign in to the site or request private, authenticated, or account-only pages.
If a request is blocked
Check the User-Agent to see whether the request came from GemmetricPublicScanner or GemmetricBot.
Know which crawler visitedCheck robots.txt, rate limits, firewall rules, and any security challenge affecting the public page.
Find the rule or settingKeep the request blocked, change the relevant rule, or contact Gemmetric when the traffic does not match what you expect.
Keep control of your siteCrawler boundaries
Gemmetric uses the pages for its website checks, not to train an AI model.
The crawlers are not used to build a general search index, resell content, or collect it for unrelated purposes.
Public checks may be blocked by robots.txt, rate limits, Cloudflare, or another WAF. Gemmetric does not promise to bypass those controls.
Crawler questions
GemmetricPublicScanner checks public pages for Free Snapshots and Prospect Audits. A site owner can block it through robots.txt, rate limits, Cloudflare, or other firewall rules.
GemmetricBot scans sites that a customer has verified or is authorized to manage. It still requests only the public pages and files needed for the website check.
GemmetricBot requests for sites a customer has verified or is authorized to manage use Web Bot Auth signatures. Supported security services can verify the signature through the public key directory advertised in the request. A User-Agent by itself should not be treated as proof because it can be copied.
Yes. GemmetricBot is listed in Cloudflare's Verified Bot Directory. The public directory entry gives site owners and supported security services an independent way to recognize Gemmetric's crawler identity.
No. Gemmetric's crawlers do not log in to the scanned website or collect private, authenticated, or account-only content.
Yes. Gemmetric retrieves robots.txt and applies the relevant published rules to the crawler identity being used.
First confirm the User-Agent, then review robots.txt, rate limits, CDN, firewall, and challenge settings. Site owners may keep the traffic blocked. If you or your client asked Gemmetric to scan the site and the behavior is unclear, contact support@gemmetric.ai.
Send us the affected URL, timestamp, User-Agent, and response details. Do not send passwords, private page content, or access credentials.