bookmark

Robots.txt meant for search engines don’t work well for web archives | Internet Archive Blogs


Description

We see the future of web archiving relying less on robots.txt file declarations geared toward search engines, and more on representing the web as it really was, and is, from a user’s perspective.

Preview

Tags

Users

  • @schneeschmelze

Comments and Reviews