If Google had to pay every web-service out there for the privilege to index their otherwise free content, that could cost Google more money than is possible for them to spend.
It would also be a disturbing direction to point the web in. The reason that google and other search engines are useful is because their bots can crawl the web like any user, they just also obey when a website asks them to stay away.
Using robots.txt or nofollow to keep bots away unless they cough up cash concerns me greatly because it means they're turning what was a "count me out please" mechanism into a paywall. A paywall that the public doesn't see, but goes up between the public and finding public content with a search engine.
It would also be a disturbing direction to point the web in. The reason that google and other search engines are useful is because their bots can crawl the web like any user, they just also obey when a website asks them to stay away.
Using robots.txt or nofollow to keep bots away unless they cough up cash concerns me greatly because it means they're turning what was a "count me out please" mechanism into a paywall. A paywall that the public doesn't see, but goes up between the public and finding public content with a search engine.