I think pay-for-verification is going to be dominant on all real-identity social media.
Reason is simple: AI
1.You can now trivially create an avatar photo, of any level of attractiveness, of any race, of any age. You can even reproduce the same character reliably in different environments/costumes via stable diffusion + LORA.
2. You can now easily create a comment history on that account, thanks to ChatGPT.
3. You can even produce voices reliably with just a few samples.
There's no real defense against AI impersonation at scale, except for charging-for-verification. The money drastically increases the cost for impersonators and scammers/catfishers, and provides resources on the other end to moderate impersonation.
I think the pay-for-verification might solve those problems ship sailed when Twitter decided that a symbol which had previously meant they'd attempted verification could be given to pretty much anyone who paid them $8 a month...
Impersonation-at-scale depends on scale, not verification of individual accounts as authentic people with authentic notability and opinions, and impersonation not at scale sometimes feels more motivated to pay the $8 than the person or position being impersonated.
I get the "i hate Twitter/elon" attitude, but the truth is that the CA system has been dependent on pay-to-verify for some time now and despite whatever grievances you have against verisign or whoever it's brought us to today safely enough.
I think the problem was not that Twitter started charging $8/month ago much as that they gave up the expensive-to-perform ID verification workflow that had been in place.
The CA system, on the other hand, has more or less abandoned the expensive, manual "extended verification" process in favor of the zero marginal cost "domain verification" process. This switchover drove the cost of a new cert to $0 for almost all users.
So I'm really not following your analogy, since Twitter gave up the workflow and started charging (aka, a cash grab), whereas CAs gave up charging and adopted a more efficient workflow (aka the expected behavior of efficient markets).
I have to disagree that it was more transparent and sensible when it meant the person had been verified. I only ever used twitter to follow bands and writers, it was useful to know it was them behind the account (or their social teams).
They muddied that water, I just don't bother with the site much anymore. I never gave a shit about the gen pop on there and it's just amplified a bunch of strangers and nobodies - I'm sure their friends and families love them, but I didn't come on the internet to see them and since twitter blue all kinds of stupid shit from nobodies with blue checks magically makes it into my feed. It's just a garbage website now.
But now (with Twitter) I can't trust anything I read without double-checking carefully it is not a fake account (and even then I cannot be sure). At least beforehand I was able to trust that Twitter has done the basic KYC.
The Meta implementation therefore seems smarter (depends on gov ID).
Why did you ever trust anything on twitter without double-checking? Consider how much misinformation you have processed over the years because of too much faith in a platform to disseminate facts over fiction.
I don't think he's talking about the information in the tweets, but the source. If I'm a fan of X and I go find X on twitter, I used to be able to more or less trust that X with a blue checkmark was the twitter account representing X. Before the hilarious shift to the gold marks, it was a lot of sifting through stupid meme accounts.
People were doing verified accounts for my ISP support twitter that day, lol
If we accept the logic that we should not trust anything posted by anyone on Twitter, then Twitter would no longer be a viable source of information whatsoever. Do you see the problem? It is poisoning the well.
someone created an account named USEmbassy_SaoTome and that had twitter verification = that was indeed the official account of the US Embassy there.
consider after Twitter Blue
someone creates an account named USEmbassy_Kiribati and has the twitter blue mark. You dont really know this is an official account. anyone can pay for this
PS: this is just hypothetical I dont really know if those embassies exists.
And the problem with pay for identity is you can buy other people's identities. The news was full impersonations right after Twitter initiated it's pay for blue check marks. That we don't here about it now may just be a matter that the news stopped caring. URL squatting was a problem for a long time - the decision that trademark holder could seize domains helped but a web address today isn't considered a solid identity at this point so "the system working" is a bit of an exaggeration.
Also, I'd note the gp didn't bad-mouth Elon or Verisign so your comment is a kind of riding a trolly false accusation.
The pay-to-verify is the reason why a significant chunk of the Internet didn't have HTTPS. It was the inception of free CAs like Lets Encrypt that made 90+% HTTPS penetration.
They already had a verification feature deployed which wasn't perfect but was reasonably strict. Then they debased it for a new revenue stream.
I don't see them making the now that we've got a bunch of people paying for it, lets reduce our revenue by suddenly becoming strict on it again decision. Or people taking the badge more seriously now it just means someone subscribed to Twitter Blue
It was an award for being an authentic notable person or entity, on the basis that the notable people and entities were most likely to be parodied or faked. Now it's an award for people that pay up
It started out as “this is who you expect it is” for things like Obama or Trump, then it became “this is a person who speaks with authority of some sort, listen to them” and now it is just “someone pays eightbux”
> “this is a person who speaks with authority of some sort, listen to them”
Rather, "this is a person that people at Twitter have blessed with authority. It's astounding that you care what we think."
> “someone pays eightbux”
To have their identity verified. This is where I pretend to be shocked that a certain class of people prefer arbitrary credential grants by unelected authorities, and that they look at a simple identity verification service that accepts anyone who is willing to show their papers and pay with absolute contempt.
> This is where I pretend to be shocked that a certain class of people prefer arbitrary credential grants by unelected authorities, and that they look at a simple identity verification service that accepts anyone who is willing to show their papers and pay with absolute contempt.
No, they think arbitrary credential grants by unelected authorities that attempted to verify people were who they said they were represented a more useful verification service than one which initially made anyone who was willing to pay $8 a "verified" account in the name of a famous person and still doesn't actually check identity.
This is where I pretend to be shocked that a certain class of people will insist a clusterfuck of such epic proportions they had to suspend it for a month after launch was actually a genius move.
It was never a pageant. It was a way for people to know they were following the famous person they wanted to follow and interact with. You're kind of scaring me that there's an actual contingent of humans that felt slighted by the notion nobody cared who they were, but people would like to know they're actually following taylor swift.
Maybe, but at least the verification seemed meaningful. And it feels like a meaningful verification doesn't generalise well: it's easy to pick a few people (arbitrarily) and truly verify them. But verifying millions of people... that's hard, unless you don't truly verify (in which case it's meaningless).
Agreed it doesn't completely invalidate it as a concept, but Elon turning the highest profile somewhat reliable implementation of an authenticity badge into a culture war artifact doesn't exactly bode well for it being a social media must-pay-for. The fact Facebook once aspired to be the platform where everyone used their real name and now can't be bothered to deactivate friendspamming sexbots without most users caring suggests that ordinary people won't exactly be queueing up to pay them $144 per annum because of their inherent trustworthiness as a verifier either.
It'd probably work a bit on LinkedIn because of the nature of the user base and lots of people already expensing Premium accounts, but funnily enough I'm not sure LinkedIn actually has that much of a fake account problem...
> but funnily enough I'm not sure LinkedIn actually has that much of a fake account problem...
Oh it does for sure, unfortunately many people are more than happy to connect with anyone who wants to connect with them because “LinkedIn doesn’t have a fake account problem”. And many people post resumes, entire employment history, all professional social connections, etc. and so all you need to do is pretend to be a student of your targets alma mater and now you have all of that personal info with the click of a Connect button. I’ve noticed trends where someone would try to connect with me and they’d be connected with some acquaintances of mine, but not once have I ever heard of that person. Those types of accounts exist, are fake, and steal information.
If you haven’t 1-1 met someone before in person or via actual business relationship you should definitely not connect with them on LinkedIn.
Most of those accounts are actual people registered with their actual name using spamming tools (some of them LinkedIn approved) to connect with as many people as possible though, usually to boost the appearance of their personal profile and have as wide a possible audience for promotional messages they share rather than to circumvent LinkedIn's information paywalls. Spamming people loosely connected with your industry with what you're doing at work is, after all, LinkedIn's intended purpose...
Oh no doubt, but I would put that in the same categorization as "fake accounts" in the sense that the intended purpose is general malfeasance. I.e. connecting not just to boost their own profile potentially but also to try and harvest any personal information you have published on the site.
> turning a $8 subscription into a culture war artifact
"Culture war artifact" is not a real term. I've never had what looks like a reasonable combination of three words come up with zero hits in a search engine.
What happened is that Musk decided to charge for identity verification instead of granting it to favored media people, and other people already involved in a culture war where Musk plays the villain added that to their long list of incomprehensible grievances.
>'"Culture war artifact" is not a real term. I've never had what looks like a reasonable combination of three words come up with zero hits in a search engine.'
Is this really your barometer for discussion, that all grammatical phrases must first exist in Google search results for them to be considered valid self-expression? If so that's pretty out there.
There's a wonderful irony here in that this a discussion about a Megacorp validating people's identity and here you are telling someone that their words are not valid because they haven't been validated by some other Megacorp.
The phrase "culture war" itself wasn't in use in English before it was used in the title of a book in the early 90s [1]. There were no search engines then.
> There's no real defense against AI impersonation at scale, except for charging-for-verification.
There is: have the state take care of that as you are a citizen that pay taxes (you already are paying for this "verification") – an ID card that contains a proof of you being... you.
But the U.S. has a phobia of any form of national ID card, so I doubt that the USDS will be able to build anything soon. It's such a shame. At least having it as an option could allow society's to start experimenting with it as a solution to these problems.
I'd like a govt service where you can verify that you're a real person but not your identity or anything of the sort. Sort of like having the third party registering with the government and then you get a token per account you make so the service can verify there isn't more than one accounts per person but they can't identify you across the services. Would also be immensely helpful useful for stopping cheating in online video games, while not requiring you to give our your real name and identity to anything
Actually the fact that it's paid makes it even more useless, because the vast majority of users will NOT be verified. I would never, ever, pay for this verification for example.
Therefore a non-verified user will not look less trustable than a verified one.
Agreed. Either they get it right (and then congrats, the Facebook account is now a passport), or they don't, and it's completely meaningless because it still means "well don't trust it though".
Who cares? Is the AI gonna RSVP to my event? Haggle over my 2nd hand item? Comment on whatever the local council has planned?
All these schemes tend to fall apart at the first contact with reality.
(Also, you are talking about "Facebook verifying users". They can't even verify who is paying them for political ads, and they certainly don't seem to care very much.)
They will definitely haggle over your second hand item. Ever tried selling an Apple product on Facebook marketplace?
You get a ton of scammers and you can't tell with some of them (until they try the actual scam, by which point you've wasted time messaging them, packaging the item, etc.)
God yes. I absolutely hate it when people cite 1984 as “what happens when you have large government.” As if no one could write a dystopian fiction novel based on whatever world view that person has.
I'm not sure, they usually work on volume. They might make 1000 fake accounts and only profit from a handful of those. They might only make a few thousand $ from those accounts, and adding $10,000 on top wipes it out entirely. Sure if they are doing some sophisticated spear fishing attack them the $10 doesn't matter, but in practice that's not what usually occurs.
Shouldn't there be a difference between "verified" and "paid"? If bad actors can just pay to have a "verified" account, then it's completely meaningless, isn't it?
Either they truly verify (which is probably hard) and I can pay to be verified, or they don't, and it's completely useless because nobody should trust the verification...
It’s not that black and white, most of these things are a mix. Some verification up front, and the normal ongoing monitoring of behavior. Putting it all up front usually presents too large a barrier to users, and causes conversion to suffer (and it’s harder to be sure than just catching bad behavior once they start using the service). Taking money makes getting banned a much larger penalty for bad actors. And that makes a lot of bad behavior unprofitable, and that subset will stop, and reduce load on their spam fighting efforts.
Right, that's interesting! Thanks for the insights :)
> and the normal ongoing monitoring of behavior
Now I get another question: say you have millions of paid, verified users (that's the goal, right?). If you can monitor millions of account successfully, then it's most likely automated, isn't it? And then it should scale to all the users, maybe? In which case again, "verified" feels meaningless (because they just improved the bot situation overall).
> And that makes a lot of bad behavior unprofitable
I wonder if the bad behavior that can be reasonably well detected is the unprofitable one. Say I can leverage the "verified" badge to scam people more efficiently -> probably I can pay 8$ for an account that will scam a few people, right? So really paying 8$ would be a deterrent for bots that purposely do disinformation, but there it's hard to know if it's a bot, isn't it? I mean many people do disinformation just out of a lack of knowledge, and that's not a reason to qualify them as bots, right?
Again, it's a mix. In many automated anti-abuse systems, there are humans in the loop for training/verification. Like, take something that's trying to catch home rental ad scammers by running basically a spam classifier on the messages they send, except trained on scammy language, like "please send the security deposit to xyz, and I'll mail you the keys". But that classifier isn't perfect, and you probably don't want false positives banning legitimate users' accounts. So instead, it just leads to a temp suspension so that if they are a scammer (high likelihood, so this is fairly safe), and drops the message in a review queue, to be verified or reversed by a human reviewer. If it reverses, that's valuable training for the automated model.
If you're charging $8/mo/account, you can justify a lot more verification than if it's just an effort to keep the user base free of scammers for nebulous brand value reasons, and there are likely to be far fewer scams to moderate anyway, since it makes it a lot more expensive/risky to ramp up 10,000 accounts, which might all get banned at once if they identify commonalities between them.
Yeah maybe. I don't know, it still feels to me that "people should not trust verified accounts because they may be scams", and bad actors can still make 10k unverified accounts.
If people learn to trust verified accounts, it becomes a risk (they shouldn't, since it is not completely reliable / it was completely bad for Twitter), right? And if they don't learn to trust verified accounts, then scammers have no reason to pay for them.
Email verification raises the cost of making a fake account, IP filtering raises the cost of making a fake account, SMS verification raises the cost of making a fake account, random algorithmic banning of new accounts raises the cost of making a fake account ...
>"There's no real defense against AI impersonation at scale, except for charging-for-verification."
What's to stop a generative AI from generating a state issued drivers license or similar to go along with that avatar photo though? Also how different is that really though from all the existing non-AI generated fake accounts?
If McDonald's required showing your driver's license to pick up your order, would it be a felony to show a fake license and pick up an order? You're not lying to a government organisation. You're just obscuring your identity to a private corporation. I fail to see how that can be any more than unethical.
Can you explain how an AI-generated fake ID for a fake social media account constitutes a felony? On what grounds exactly? Facebook despite attempting to normalize the handing over of government-issued identification to an advertising platform, is not a government entity.
I don't know enough about your legal system to be sure, but just as an option, wouldn't it be possible that the mere act of creating a my fake ID could become a felony, if it isn't already?
Just like the act of defacing your currency is illegal, even if you don't pay with that money.
For me, the safest verification is what I had to do for my first email address in Germany, which is that the postman comes to your house and verifies that you are who you claim to be. Alternatively, and pretty much equally secure, our passports include the option of digitally verifying your identity at home. However, I'm not willing to do either that for Facebook, the last option is probably not possible with US passports, and the first option would be rather expensive.
I have 3 distinct pieces of ID. Both my ID card and my passport should theoretically be usable to verify my identity online. We also have a digital only[0] tool at our disposal. It’s insane to me that we still don’t have a reliable way to verify an identity online.
You are not generating a real physical ID but a representation of an ID, a jpeg. Nor would you be using that representation of an ID to identify and actual person for something that legally requires proof of ID like purchasing alcohol, opening a bank account or driving a car. The sole context for this representation is for a fake persona that exists solely on a social media service.
While it is true that using a fake ID to commit other crimes is generally more serious and is the more common case, producing an image of a fake ID is still recognized as a felony.
"There's no real defense against AI impersonation at scale,"
Defamation liability. I'm not sure the laws have worked out yet, but I suggest that these companies may be in for some lawsuits if thing go awry, and they have not shown to be actively working against it.
> Most people use social media to keep track of people they already know.
And most people who drive for Uber are just doing so to make a few bucks in their daily life. And most people on AirBnb are renting their spare room or couch.
/s
Most people follow a handful of a famous people or companies, and some friends. Especially on twitter, which was very "one to many" following relationships. Thats why verification was so impactful, it gave the "one" in "one to many" something special, and a way to establish authenticity.
Reason is simple: AI
1.You can now trivially create an avatar photo, of any level of attractiveness, of any race, of any age. You can even reproduce the same character reliably in different environments/costumes via stable diffusion + LORA. 2. You can now easily create a comment history on that account, thanks to ChatGPT. 3. You can even produce voices reliably with just a few samples.
There's no real defense against AI impersonation at scale, except for charging-for-verification. The money drastically increases the cost for impersonators and scammers/catfishers, and provides resources on the other end to moderate impersonation.