If Google had to pay every web-service out there for the privilege to index their otherwise free content, that could cost Google more money than is possible for them to spend.
Then again, Twitter and Facebook are certainly large enough that some kind of deal could have been negotiated. But if they did that, would other social networks uproar in protest because now it's not just Google monopolizing their power.. it's the 3 largest websites out there doing it together.
If Google had to pay every web-service out there for the privilege to index their otherwise free content, that could cost Google more money than is possible for them to spend.
It would also be a disturbing direction to point the web in. The reason that google and other search engines are useful is because their bots can crawl the web like any user, they just also obey when a website asks them to stay away.
Using robots.txt or nofollow to keep bots away unless they cough up cash concerns me greatly because it means they're turning what was a "count me out please" mechanism into a paywall. A paywall that the public doesn't see, but goes up between the public and finding public content with a search engine.
Then again, Twitter and Facebook are certainly large enough that some kind of deal could have been negotiated. But if they did that, would other social networks uproar in protest because now it's not just Google monopolizing their power.. it's the 3 largest websites out there doing it together.