It’s not that black and white, most of these things are a mix. Some verification up front, and the normal ongoing monitoring of behavior. Putting it all up front usually presents too large a barrier to users, and causes conversion to suffer (and it’s harder to be sure than just catching bad behavior once they start using the service). Taking money makes getting banned a much larger penalty for bad actors. And that makes a lot of bad behavior unprofitable, and that subset will stop, and reduce load on their spam fighting efforts.

Right, that's interesting! Thanks for the insights :)

> and the normal ongoing monitoring of behavior Now I get another question: say you have millions of paid, verified users (that's the goal, right?). If you can monitor millions of account successfully, then it's most likely automated, isn't it? And then it should scale to all the users, maybe? In which case again, "verified" feels meaningless (because they just improved the bot situation overall).

> And that makes a lot of bad behavior unprofitable

I wonder if the bad behavior that can be reasonably well detected is the unprofitable one. Say I can leverage the "verified" badge to scam people more efficiently -> probably I can pay 8$ for an account that will scam a few people, right? So really paying 8$ would be a deterrent for bots that purposely do disinformation, but there it's hard to know if it's a bot, isn't it? I mean many people do disinformation just out of a lack of knowledge, and that's not a reason to qualify them as bots, right?

Again, it's a mix. In many automated anti-abuse systems, there are humans in the loop for training/verification. Like, take something that's trying to catch home rental ad scammers by running basically a spam classifier on the messages they send, except trained on scammy language, like "please send the security deposit to xyz, and I'll mail you the keys". But that classifier isn't perfect, and you probably don't want false positives banning legitimate users' accounts. So instead, it just leads to a temp suspension so that if they are a scammer (high likelihood, so this is fairly safe), and drops the message in a review queue, to be verified or reversed by a human reviewer. If it reverses, that's valuable training for the automated model.

If you're charging $8/mo/account, you can justify a lot more verification than if it's just an effort to keep the user base free of scammers for nebulous brand value reasons, and there are likely to be far fewer scams to moderate anyway, since it makes it a lot more expensive/risky to ramp up 10,000 accounts, which might all get banned at once if they identify commonalities between them.

Yeah maybe. I don't know, it still feels to me that "people should not trust verified accounts because they may be scams", and bad actors can still make 10k unverified accounts.

If people learn to trust verified accounts, it becomes a risk (they shouldn't, since it is not completely reliable / it was completely bad for Twitter), right? And if they don't learn to trust verified accounts, then scammers have no reason to pay for them.

"This scam is verified by Meta as being legitimate"