Platform Broking The IWE Benefits Sectors Resources Tools How It Works About Book a call
Home › Benefits Explorer › Our research bot
AlltoogetherBenefitsResearch

Our research bot

If you have seen it in your logs

AlltoogetherBenefitsResearch is the crawler Alltoogether uses to read what UK employers publish about staff benefits: holiday, pension, parental leave, health cover and the rest. It reads public careers pages and job adverts, slowly, and only where your robots.txt allows.

This page explains what it collects, how it behaves, and how to stop it. You can block it in one line, or ask us by email and we will stop.

Last reviewed 2 November 2026Follows robots.txtOne request a second at mostStops when asked

What it collects

It reads

  • Public careers, jobs and benefits pages on employers' own websites.
  • Public job adverts from job-board feeds that are published for reuse, and job pages published for search engines.
  • Public registers: the GOV.UK gender pay gap service, which lists employers and the website each one filed.

It never

  • Logs in, fills in forms or uses keys it was not given.
  • Gets round a block, a challenge page or a refusal.
  • Runs scripts, downloads files or reads more than 3 MB from a page.
  • Visits private or internal network addresses.
  • Keeps names, email addresses or phone numbers: they are removed before anything is stored.

From each page it keeps facts about benefits (for example, "25 days' holiday plus bank holidays" or "5% employer pension contribution"), each with the short sentence it came from, the link and the date, so anyone can check it. Raw page text is never published. We keep it so we can re-check each fact when our reading improves, and we delete it for any employer that asks to be removed.

How it behaves

User agentAlltoogetherBenefitsResearch/0.3 (+https://www.alltoogether.com/bot)
robots.txtRead before every site and followed, including rules for User-agent: * and any group naming us. If your robots.txt redirects somewhere it may not go, it leaves your whole site alone.
SpeedAt most one request a second to any site, and a small daily cap per site. Most sites see fewer than fifteen requests in a visit.
If you say noAny "too many requests" or "forbidden" answer stops it on your site for at least 24 hours. A site that refuses it is left alone.
RedirectsChecked before they are followed. It never follows a redirect to a site it has been told not to visit.
How oftenNot on a schedule. It runs when we update the UK Benefits Explorer.

How to stop it

In robots.txt. Add these two lines. It reads your robots.txt before every visit, so this takes effect next time.

User-agent: AlltoogetherBenefitsResearch
Disallow: /

By email. Write to hello@alltoogether.com with the subject "Benefits research opt-out" and your domain. We add it to our opt-out list, which the bot checks before every request, and we remove what we hold about your organisation from the UK Benefits Explorer.

Something wrong? If a fact about your organisation is wrong or out of date, email us and we will correct it.

What we use it for

The facts power our UK Benefits Explorer, which shows employers what is common in their sector and which services their existing cover usually includes. Every fact is shown with its quote and a link back to the page it came from.

We do not sell the data. Not mentioned is not the same as not offered: the UK Benefits Explorer only shows what an employer's public pages say.

Not sure it was us?

Anyone can put our name in a user agent. If you see AlltoogetherBenefitsResearch behaving differently from this page, email hello@alltoogether.com with the times and addresses from your logs and we will check.