The bank example really needs to die, its not reflective of what people often use databases for.
1. Bank transactions are easy to express as a join-semilattice[1], which means its intrinsically easy to build a convergent system of bank accounts in a distributed environment. Many problems are not easily expressible as these and usually trade off an unacceptable amount of space to formulate as a join-semilattice and still work with complicated types of transactions.
2. A bank transaction has a small upper bound on the types of inconsistency that can occur: at worst you can make a couple overdrafts.
3. A bank transaction has a trivial method for detecting and handling an inconsistency. Detection is easy because its just a check that the transaction you're applying doesn't leave the account with a negative balance. Handling is easy, because the bank can just levy an overdraft charge on an account holder (and since they have the ability to collect debts on overdrawn accounts, this easily works for them). Complicated transactions often don't have a trivial method of detecting inconsistency and handling inconsistency is rarely as simple as being able to punish the user. In fact corrective action (and likewise, inconsistency) is often non-intuitive to users because we expect causality in our interaction with the world.
> The real world rarely involves having a consistent view of anything
I strongly disagree with this. Most of our interaction with the world is perceived consistently. When I touch an object, the sensation of the object and the force I exert upon the object is immediate and clearly corresponds to my actions. There is no perception of reordering my intentions in the real world.
We expect most applications to behave the same way, because the distributed nature of them is often hidden from us. They present themselves as a singular facade, and thus we typically expect our actions to be reflected immediately and in the order we issue them.
Furthermore, this availability vs consistency dichotomy is an overstatement of Brewer's theorem. For instance, linearizable algorithms like Raft and Multi-Paxos DO provide high availability (they'll remain available so long as a majority of nodes in the system operate correctly). In fact, Google's most recent database was developed to be highly available and strongly consistent [2].
"Bank transactions are easy to express as a join-semilattice[1],"
That may be true for a given set of transactions in a very abstract way, but that has some serious practical problems. First, like it or not, banks have a history of making transactions non-associative and non-commutative; there are many reported cases from the wild of banks deliberately ordering transactions in a hostile manner so that the user's bank account drops below the fee threshold briefly even though there is also a way to order the transactions so that doesn't happen. So as a developer, telling your bank management that you are going to use a join-semilattice (suitably translated into their terms) is also making a policy decision, a mathematically strong one, about how to handle transactions that they are liable to disagree with.
There is also the problem of variable time of transmission of transactions meaning that you may still need to have some slop in the policies, in a way that is not going to be friendly to the lattice.
The nice thing about join semilattices and the similar mathematical structures is that you get some really nice properties out of them. Their major downside, from a developer point of view, is that typically those properties are very, very, very fragile; if you allow even a slight deviation into the system the whole thing falls apart. (Intuitively, this is because those slight deviations can typically be "spread" into the rest of the system; once you have one non-commutative operation in your join-semilattice it is often possible to use that operation in other previously commutative operations to produce non-commutative operations... it "poisons" the entire structure.)
I think you're badly underestimating the difficulty of the requirements for anything like a real bank system.
Apologies, you're right, I've never implemented bank accounting in the real world, I'm only talking about it in the sense of an account balance as a "toy problem", where its often used to discuss the virtues of weak consistency by being able to use overdraft charges.
Whether or not its actually implemented with something like CRDTs in the real world isn't really my point, my point is that its a cherry-picked example of a system that clearly does reorder operations and has weaker-than-usual concerns about witnessing inconsistency.
> There is no perception of reordering my intentions in the real world.
Never tried to type something by thinking only of the word (rather than specifically the sequence of key strokes) only to have your fingers execute the presses in slightly the wrong order -- giving you something like "teh" instead of "the"? I have, a lot actually, and if I'm in a hurry or distracted, it can even pass by unnoticed, leaving me with the impression I typed the correct word while having actually executed the wrong sequence. (Hurry or distracted I think would be the most analogous to systems which don't take the time to register aggregate feedback and verify correct execution after the fact.)
Humans have at least one experience of trying to execute a distributed sequence of actions and having the output be mangled by the actual ordering events so common they invented a word for it -- the "typo".
It's insanely easy to explain to people that sometimes really complicated systems make a typo when they're keeping their records, because it's such a shared, common experience.
It's a cool metaphor, and I'm sure there are occasions where its easy to pass this off as innocuous. But inconsistent semantics can produce blemishes and bugs that can be difficult to anticipate and can very negatively impact user experience if you aren't careful. The typo explanation is acceptable when I just see some comment I posted being reordered after a page refresh. It's less so if I get locked out of my account because my password change hasn't fully replicated to all servers (and even less so if my consistency model doesn't converge).
To be clear: there's an economic tradeoff to make here that isn't identical at all scales. At a certain point, lost revenue due to user dropoff from availability is more expensive than users who stop using a site due to a frustrating experience (and sometimes that scale itself is the compelling feature able to stave off a user from giving up on your app completely). But in most cases, I think its advisable to start with stronger consistency semantics and only weaken them when a need presents itself. Hell, Spanner is the foundation for Google's ad business, if that's not living proof that you can preserve strong consistency semantics and keep your system sufficiently available, I'm not sure what is.
> The typo explanation is acceptable when I just see some comment I posted being reordered after a page refresh. It's less so if I get locked out of my account because my password change hasn't fully replicated to all servers (and even less so if my consistency model doesn't converge).
The typo model is merely meant to explain why it happened. You'd be no more happy if you were locked out of a door because a locksmith fat-fingered the code on setting a pin. Not all typos are harmless, and that wasn't the point of my analogy.
Helping people understand why these technology mistakes happen -- through intuitive models like typos -- helps us discuss when mistakes are and aren't acceptable, and how we're going to address the cases where they're not. It simply gives me a way to discuss it with non-technical people: under what situations would a person doing the computer's job making a typo be catastrophic instead of merely annoying?
Most managers or clients can understand that question, at an intuitive level.
> But in most cases, I think its advisable to start with stronger consistency semantics and only weaken them when a need presents itself.
Completely agree! Unless you have a specific reason it's necessary for you, you likely don't have to actually address the inconsistency directly and are better off letting the database handle that for you.
Oh apologies, I misunderstood what you were saying! Yes, agreed, this is a good method for communicating what's occurring in weakly consistent systems to stakeholders. Definitely going to use this in the future.
Do you (or anyone) know of a better example than the bank account? I wouldn't be against changing it.
I wanted to communicate through a fairly succinct and understandable example that there are potentially high stakes, and I wanted to show a 'NoSQL' example rather than a micro-services one. It was the first one that came to mind.
Apologies, I didn't mean it as a critique of your post. What I meant was that its often used as a strawman example of why you don't need consistency[1]. Bank transactions are extremely simple compared to the majority of things we build on top of databases.
Also, banks have very low consistency requirements.
Almost every transaction is of the form: 1. estimate authorization; 2. present request for settlement; 3. settle during batch processing; 4. any additional resolution steps (eg, returned checks)
The estimator used in 1 need not be fully consistent because a) as long as the periods of inconsistency are low (eg, < 10 sec) it won't affect the usage patterns of most accounts and b) the bank only need quantify the risk to their system (and be less than the cost of improving the system), rather than have it be risk-free. Risk management techniques like preventing rapid, high-value transactions on top of a basic estimated value like "available balance" is fine for the estimation used in authorization phase.
The actual settlement is run during a special batch transaction during hours where they can't receive new transactions, and thus is a special case of it being really easy to implement consistency of the outcome, and it's acceptable if the settlement has some number of accounts end up negative or requiring additional steps (such as denying the transaction to the presenting bank), as long as the number isn't too high, eg, the estimator in step 1 is right most of the time.
Basically, the only requirement the banks have is that a) they can estimate how inconsistent or wrong their system is until their batch settlement where they resolve the state and b) the number of accounts which will be in error when settled remains low. Then they just hire people to sort out some of the mess and eat some amount of losses as easier than trying to make the system perfect.
The fact is that banks make better use of what databases are and aren't than most designs, because they're very clear about when and where they want consistency (weekdays, 6am Eastern in the US) and the (non-zero!) rate at which the system can be in error when it resolves the state. Any application that can be clear about those can get good usage of databases.
> The estimator used in 1 need not be fully consistent because a) as long as the periods of inconsistency are low (eg, < 10 sec) it won't affect the usage patterns of most accounts
A lot of transactions that need to be reconciled happens without involving 1. e.g. checks and offline card transactions. Even ATMs can often continue to dispense cash when their network is offline, though there arguable 1 applies in the form of maximum withdrawal limits.
But in general I agree with you - their system is built around having accepted, calculated and accounted for the risks rather than try to mitigate every possible risk. Which makes sense because a large proportion of a banks business is to understand and manage risk, and some risks are just happens cheaper to accept than to mitigate via technology or business processes (and the ones that aren't, or aren't any more, are where you'll see banks push new technology, such as making more settlements happens faster and fully electronically)
> I strongly disagree with this. Most of our interaction with the world is perceived consistently. When I touch an object, the sensation of the object and the force I exert upon the object is immediate and clearly corresponds to my actions. There is no perception of reordering my intentions in the real world.
You misunderstand me, and I appreciate that I should have been clearer: A single observer in a single location that is both the sole observer and creator of event will have a consistent view. But this isn't a very interesting scenario - the type of interactions that matter to us are operations that requires synchronisation between at last two different entities separated in space.
Consider if you have two identical objects, on opposite sides of the world, that represents the same thing to you and another person. When you move one, the other should move identically.
Suddenly consistency requires synchronisation limited by the time it takes to propagate signals between the two locations to ensure only one side is allowed to move at once.
This is the cost of trying to enforce consistency. If you lose your network, suddenly both or at least one (if you have a consistent way of determining who is in charge) are unable to at least modify, and possibly read the state of the system, depending on how strict consistency you want to enforce. In this specific case, it might be necessary.
But most cases rather involves multiple parties doing mostly their own things, but occasionally needing to read or write information that will affect other parties.
E.g. consider a forum system. It requires only a bare minimum amount of consistency. You don't want it to get badly out of sync, so that e.g. there's suddenly a flood of delayed comments arriving, but a bit works just fine. We have plenty of evidence in the form of mailing lists and Usenet, which both can run just fine over store-and-forward networks.
Yet modern forum systems are often designed with the expectation of using database transactions to keep a globally consistent view. For what? If the forum is busy, the view will regularly alrady be outdated by the time the page has updated on the users screens.
> Furthermore, this availability vs consistency dichotomy is an overstatement of Brewer's theorem. For instance, linearizable algorithms like Raft and Multi-Paxos DO provide high availability (they'll remain available so long as a majority of nodes in the system operate correctly). In fact, Google's most recent database was developed to be highly available and strongly consistent [2].
They'll remain available to the portion of nodes that can reach a majority partition. In the case of a network partition that is not at all guaranteed, and often not likely. These solutions are fantastic when you do need consistency, in that they will ensure you retain as much availability as possible, but they do not at all guarantee full availability.
Spanner is interesting in this context because it does some great optimization of throughput, by relying on minimising time drift: If you know your clocks are precise enough, you can reduce synchronisation needed by being able to order operations globally if timestamps are further apart than your maximum clock drift, which helps a lot. But it still requires communication.
In instances where you can safely forgo consistency, or forgo consistency within certain parameters, you can ensure full read/write availability globally (even in minority partitions) even in the face of total disintegration of your entire network. But more importantly, it entirely removes the need to limit throughput based on replication and synchronisation speeds. This is the reason to consider carefully when you actually need to enforce consistency.
> You misunderstand me, and I appreciate that I should have been clearer: A single observer in a single location that is both the sole observer and creator of event will have a consistent view. But this isn't a very interesting scenario - the type of interactions that matter to us are operations that requires synchronisation between at last two different entities separated in space.
My point in bringing that up is that applications don't present the appearance of appearance of entities separated in space, that's why weak consistency is unintuitive. We expect local realism from nearby objects the same as we expect behavior in an application that exhibits effects from our immediate actions.
> E.g. consider a forum system. It requires only a bare minimum amount of consistency. You don't want it to get badly out of sync, so that e.g. there's suddenly a flood of delayed comments arriving, but a bit works just fine. We have plenty of evidence in the form of mailing lists and Usenet, which both can run just fine over store-and-forward networks.
Most threaded or log-like things are examples that fare well with weak consistency (because they're semi-lattices with lax concerns about witnessing inconsistency). I'm not saying that there aren't examples where its a tradeoff worth making, I'm saying that many applications are built on non-trivial business logic that coordinates several kinds of updates where it is tricky to handle reordering.
> They'll remain available to the portion of nodes that can reach a majority partition. In the case of a network partition that is not at all guaranteed, and often not likely. These solutions are fantastic when you do need consistency, in that they will ensure you retain as much availability as possible, but they do not at all guarantee full availability.
Weakly consistent databases can't guarantee full availability either when a client is partitioned from them. In fact there's no way to guarantee full availability of internet services to an arbitrary client. I think the importance of network partitions is somewhat overstated by HA advocates anyways. They're convenient in distributed systems literature because they can stand-in for other types of failures, but link failures (and other networking failures, but link hardware failures are disproportionately responsible for networking outages[1]) happen at an order of magnitude less than disk failures.
> In instances where you can safely forgo consistency, or forgo consistency within certain parameters, you can ensure full read/write availability globally (even in minority partitions) even in the face of total disintegration of your entire network. But more importantly, it entirely removes the need to limit throughput based on replication and synchronisation speeds. This is the reason to consider carefully when you actually need to enforce consistency.
I'm all for giving careful consideration to your model. And I agree, there are several cases where weakening consistency makes sense, but I think they are fairly infrequent. Weakening consistency semantics per operation to the fullest extent can add undue complexity to your application by requiring some hydra combination of data stores, or come at a large cost by requiring you to roll your own solution to recover some properties like atomic transactions. You called this "laziness" earlier, but I think this is actually pragmatism.
1. Bank transactions are easy to express as a join-semilattice[1], which means its intrinsically easy to build a convergent system of bank accounts in a distributed environment. Many problems are not easily expressible as these and usually trade off an unacceptable amount of space to formulate as a join-semilattice and still work with complicated types of transactions.
2. A bank transaction has a small upper bound on the types of inconsistency that can occur: at worst you can make a couple overdrafts.
3. A bank transaction has a trivial method for detecting and handling an inconsistency. Detection is easy because its just a check that the transaction you're applying doesn't leave the account with a negative balance. Handling is easy, because the bank can just levy an overdraft charge on an account holder (and since they have the ability to collect debts on overdrawn accounts, this easily works for them). Complicated transactions often don't have a trivial method of detecting inconsistency and handling inconsistency is rarely as simple as being able to punish the user. In fact corrective action (and likewise, inconsistency) is often non-intuitive to users because we expect causality in our interaction with the world.
> The real world rarely involves having a consistent view of anything
I strongly disagree with this. Most of our interaction with the world is perceived consistently. When I touch an object, the sensation of the object and the force I exert upon the object is immediate and clearly corresponds to my actions. There is no perception of reordering my intentions in the real world.
We expect most applications to behave the same way, because the distributed nature of them is often hidden from us. They present themselves as a singular facade, and thus we typically expect our actions to be reflected immediately and in the order we issue them.
Furthermore, this availability vs consistency dichotomy is an overstatement of Brewer's theorem. For instance, linearizable algorithms like Raft and Multi-Paxos DO provide high availability (they'll remain available so long as a majority of nodes in the system operate correctly). In fact, Google's most recent database was developed to be highly available and strongly consistent [2].
[1]: https://en.wikipedia.org/wiki/Semilattice
[2]: http://research.google.com/archive/spanner.html