What it does
AvaReply lets businesses put an AI assistant on their website. To answer accurately, the assistant needs to know what the website says, so AvaReplyBot reads the site's pages and turns them into a knowledge base for that assistant only.
When it visits
AvaReplyBot doesn't browse the web on its own. It only visits a site after an AvaReply customer adds that website to their assistant. Customers add their own websites.
After that first read, it can come back to refresh the content, either when the customer asks for it or on a schedule they choose (daily, weekly or monthly).
What it reads
- Only public pages: the same pages anyone can open without logging in. It doesn't fill in forms or log in to anything.
- Only the website that was added. It follows links within that site and can use its
sitemap.xml, up to a set limit on pages and link depth. - It doesn't use pages marked
noindex. - Some sites only show their content after JavaScript runs. For those pages it may use a headless browser, which sends the same user agent.
robots.txt is always respected
Before each crawl, AvaReplyBot reads your /robots.txt and follows its rules for AvaReplyBot. If there's no section for AvaReplyBot, it follows the rules for *. AvaReply customers can't turn this off.
To block AvaReplyBot from your whole site, add this to robots.txt:
User-agent: AvaReplyBot Disallow: /
To keep it out of certain sections only:
User-agent: AvaReplyBot Disallow: /private/ Disallow: /account/
robots.txt is read at the start of every crawl, so changes apply from the next crawl onward.
Contact
Questions about AvaReplyBot, or want content from your site removed? Email bot@example.com and include your domain.