I’ve written before about my desire for more transparency at every level of government. There are very few opportunities to push this locally, but every time one comes up, it’s important to do so – and one such opportunity came in the last month.
A resident told me she had been mislead by something that had previously been on Camden’s website, but that has since been changed. My usual way to find such changes would be using the Internet Archive. But when I looked there, it hadn’t archived it. In fact, it had barely archived any of Camden’s website.
I’m pretty technologically savvy, but I’m not a web designer (Exhibit A: this website). Nonetheless, I know that ain’t normal – and it’s usually done because the website orders the Internet Archive not to archive its files with a file called robots.txt. This file tells web crawlers (also called robots, bots, or spiders) whether they can access all, some, or none of the site.
There are legitimate uses of this, e.g. http://www.gov.uk/robots.txt is limited in its application and explains line-by-line why it has excluded each web crawler. But unless each restriction can be explained, public authorities’ websites should be open to all and free to archive – it’s public information, after all.
However, this is what Camden’s robots.txt file said:
In English, this means Google can access the website, but the vast majority of it – the stuff in the folder ‘includes’ is off-limits to all other web crawlers. That includes the Internet Archive, but also Bing, Yahoo!, etc. This inherently limits people’s ability to find the right information on Camden’s website, especially if you’re looking for former pages or policies, as I was.
I requested that Camden change this or give a reason why not. They’ve responded today saying that they’ve removed this restriction altogether.
This sounds really minor – and, truth be told, it is – but it’s indicative of the sort of limitations on government transparency that are easy to fix and that could cumulatively make it a lot easier to hold all levels of government to account.