Our crawler visits a website only at the request of that site's own owner — as part of building the InstantAIguru customer-service AI agent for their business. It is not a search, training, or data-collection crawler for the open web.
Every request carries this User-Agent:
Mozilla/5.0 (compatible; InstantAIGuruBot/1.0; +https://instantaiguru.com/bot)Requests are cryptographically signed using Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519). Each request includes Signature-Agent, Signature-Input, and Signature headers, so you can verify our identity by signature rather than trusting the User-Agent string alone.
Our public keys are published as a signed JSON Web Key Set at:
https://instantaiguru.com/.well-known/http-message-signatures-directoryThe Signature-Agent value in our requests is "https://instantaiguru.com".
| robots.txt | Honored by default, including Crawl-delay. We identify as InstantAIGuruBot for rule matching. |
|---|---|
| Scope | We crawl only domains operated by the customer who commissioned the agent — which may be more than one, when that customer runs multiple sites. |
| Rate | Conservative crawl rate; we back off on 429/503 responses. |
| Owner override | A site owner who has authorized us may ask us to crawl content their robots.txt would otherwise exclude — only ever with their explicit consent. |
To ask a question, request that we stop crawling, or report unexpected behavior, email [email protected]. You can also block InstantAIGuruBot in your robots.txt at any time and we will comply.