IMHO boring tech is great because it lets you focus on the actual tech of your product. I run a SaaS app and I'd like to think we do alot of cutting edge things in various places, as it relates to the actual product. For things that are "behind the scenes" like databases, backend fraemworks, etc, etc, I prefer to keep all that stuff as boring and as stable as possible. For me working solo on a project, my time is very limited. So I would much rather work on interesting new features for my product than having to touch crap that my customers don't care about. Because at the end of the day my customers don't know and don't care that I use Node vs Deno or Bun, or that I use NPM instead of pnpm, or that I'm not on the latest version of node or Postgres. They don't know all that but they do know how well my app works or what features it has.
Exactly - This is known to graybeards and mostly ignored by the youths.
We've seen the wheel re-invented many times and would prefer to work on something other than the wheel again. Stuff like solving user problems and making money.
Meanwhile you have the coworker who uses some new but soon to be deprecated language/framework on every project, leaving a field of unsupportable debris in their wake..
I experienced this few years ago. I was hired with a team of consultants (I know.. I know..) because the system in question would have data randomly disappear from their databases every few weeks and it would take the operation team a month to notice, so by then the issue was harder to rectify.
The engineering team in question had proposed to rewrite an old Java app with AWS Amolify and replace their Postgres database with Dynodb. The whole thing was then ducktaped with Lambdas and spread across multiple regions. They had not bothered writing a single line of test, never mind having any kind of build and deployment pipeline.. they didn’t even have basic error reporting.
After doing a deeper dive, we discovered that engineers didn’t have a local environment but only had access to staging, however they could easily connect to the production database from local by just updating environmental variables. It turns out that one of them had forgotten to switch it back after debugging something on production from his machine and had forgotten to switch it back.. he had a background cleanup job running in intervals which was wiping data..
It was a complete nightmare, the schema less nature of Dynamo made it harder to understand what is happening and the React UI would crash every 15 minutes due to an infinite loop.
The operation team had learned to use the Chrome console and clear the local storage manually before the window feeezes..
Being charitable, in order for one of those new technologies to become mature and boring it requires guys like that to actually use it for things. So while it might be misguided we thank them for their sacrifice of getting caught on every sharp edge so they might be dulled (or at least documented) later.
And if the new tech is beneficial and adds enough value that it makes it worth replacing the old tech, then by all means go for it.
However, I can't count many companies I've seen decide to get into "the cloud" only to do lift-and-shifts and are now running their stack in slower and more expensive ways.
Over the last decade I worked for a fintech that did analytics for the investment banking industry and between 2016-2020 the amount of people that were shocked we weren't trying to shove blockchain somewhere was surreal.
Oh yeah, SOMEONE has to be be first.
It just doesn't have to be YOU!
If you want to be successful in your career, when you are put in charge of a big new project, on a tight timeline, high management visibility, etc.. you dig into your existing tried & true toolkit to get the job done. There's so many other variables, why needlessly add more risk no one asked for?
But yes, I'm glad there are maniacs out there.. I just don't want to work with them.
Granted, I remember symfony was very good, even in versions 1.x. It truly made working with PHP not suck as much as it did back in the day. I don't remember if there was ever a product of which it was born.
> you dig into your existing tried & true toolkit to get the job done
Haha. Yes. But when the c-suite is made up of top level management pushed out of the s&p 500, they always assume it’s their tried and true toolkit from another company. Believe me, it’s never the hammer the current engineering staff is holding. I’m slow clapping so hard for the business school graduates right now…
I've seen projects akin to "Excel sucks because its written in X, I am very smart, we will rewrite it to run in browser only with this new framework I read about, backed by microservices in Kotlin/Rust/some new fad, running server less on AWS (using this new thing thats just out of beta that I saw at re:Invent)".
And we need to stop all new feature work on Excel since its legacy, so give me 80% of the dev team to do above. Oh btw, they don't know the alphabet soup of stuff I decided to use, so we will also start firing them as well, as I need to hire for these special skills.
Eventually you need to admit COBOL is dead and rewrite. Eventually some library/framework is dead and you need to rewrite. However there are lots of options, and often your best is in place rewrite just small parts at a time, slowly getting rid of the legacy code over a couple decades - there is no real hurry here.
Eventually styles will change and you will have to redo the UI. This will happen much more often than the above. Your program may look very different but if you have a good architecture this is a superficial change. It may still be expensive, but none of your core logic changes. Normally you keep the old and new UI running side by side (depending on the type of program may be different builds, other times it is just a front end) until you trust the new one. (depending on details it may be an all at once switch or one screen/widget at a time)
Just because it is populate doesn't mean it isn't dead. Or should be dead. COBOL was really innovated in the day, but many of those innovations proved to be bad ideas. However switching to something else is very hard and expensive - thus it continues on.
Well, I'm not sure I'd say Windows is a good example of that anymore, in fact I was going to use it to argue the very point: as they seem to rewrite the UI in new frameworks, we've lost a lot of features (not even talking about speed and reliability)
I can, off the top of my head, name at least half a dozen of "those guys" and describe in detail the wreckage they left behind. Including one that was a major contributor to an 80%-of-the-billings client finding an alternative agency resulting in almost 100 people losing their jobs. :sigh:
Your phrasing implies that the Greenfield rewrite is always a mistake.
Depending on the project, it might actually be the only sane option if you're still required to make significant changes to the application and features have be be continuously added - and the project has already become so full of technical debt that even minor changes such as mapping an additional attribute can take days.
As an easily explained example for such: I remember an angular fronted I had to maintain a while ago.
The full scope of the application was to display a dynamic form via multiple inputs and interdependent on the selected choices of the available form (so if question 1 had answer a, question 2+3 had to be answered etc).
While I wouldn't call such a behavior completely trivial, on a difficulty line it was definitely towards the easy end - but the actual code was so poorly written that any tiny change always led to regressions.
It was quite silly that we weren't allowed to invest the ~2 weeks to recreate it
Another example that comes to mind is a backend rewrite of a multi million PHP API to a Java backend. They just put the new Java backend in front of the old API and proxied through while they implemented the new routes. While it took over 2 years in total, it went relatively well.
But yeah, lots of Greenfield rewrites end in disaster. I give you that!
My company spent $billion and several years in a rewrite of a core product a few years ago.
I'm convinced on hindsight that we could have just refactored in place and been just as well off. Sure there would be some code that is still the ugly mess that made us jump to the big rewrite in the first place. However we would have had working code all along to ship. Much more importantly, we fixed a lot of problems in the rewrite - but we introduced other problems that we didn't anticipate at the same time, and fixing them means either another rewrite, or do an in-place refactor. The in-place refactor gives one advantage - if whatever new hotness we choose doesn't work in the real world we will discover before it becomes a difficult to change architecture decision.
A few years after a greenfield rewrite, the codebase is going to go back to the same level of quality it was before the rewrite, because the level of code quality is a function of the competency of the team, not the tech stack.
The only time it really makes sense to do a rewrite, is when either a new architecture/technology is going to be used that will impact team competency, or the team's competency has improved significantly but is being held back by the legacy application.
In both of those situations though, you could and should absolutely cut the application into pieces, and rewrite in place, in digestible, testable chunks.
The only time it makes sense to do a wholesale greenfield rewrite, is political. For example, maybe you have a one-time funding opportunity, with a hard deadline (rewrite before acquisition or ipo, etc).
I think it is safe to say we have improved as a company between when the code was started and the rewrite. And we have improved a lot since then.
We have also improved a lot as an industry. The rewrite was started in C++98 because C++11 was still a couple years away. Today C++23 has a lot of nice things, but some of the core APIs still are C++98 (though less every year) because that is all we had the ability to use. And of course rust wasn't an option back then, but if we were to start today would get a serious look.
Once we did a major rewrite from Perl/C++/CORBA into Java, during the glory days of Java application servers, three years development that eventually went nowhere, or maybe it did, who knows now.
In hindsight, cleaning up that Perl and C++ code, even where both languages stand today, would have been a much better outcome, than everything else that was produced out of that rewrite.
But hey, we all got to improve our CVs during that rewrite and got assigned better roles at the end, so who cares. /s
> Another example that comes to mind is a backend rewrite of a multi million PHP API to a Java backend. They just put the new Java backend in front of the old API and proxied through while they implemented the new routes. While it took over 2 years in total, it went relatively well.
Their next example was exactly what you asked for, 2 years rewrite.
Bonus points from me because they didn't wait for the whole rewrite to be done, and instead started using the new project by replacing only parts of the original one.
If you are not architectured around the bridge it is really hard to add it latter. Probably the biggest advantage of micro services is they have built in bridges. Monoliths often have no obvious way to break things up. Every time you think you want to you discover some other headache.
Totally agree, but how often is the young coworker using the hip new language in their position because they were told learning it was the key to getting a job in a competitive market and it is what they have to fall back on?
Somewhat similarly, I feel like the boring/mature infra often gets ripped up in favor of something hip and new by a CIO who wants a career checkmark that they "modernized" everything. Then they move on to the next company and forget the consequences of breaking what was stable.
Modernization can mean replacing 50 year old technology with "20 year old" technology. I stuck in the quotes because something that is perceived to be that old is probably under active development, and modern while remaining boring.
There are different 50 year old technologies. Fortran is older than that, but still going strong in some niches because for a few niches it works good enough. (for some numeric operations it compiles to the fastest code and that matters). COBOL is older than that too, but everyone is replacing it because nobody wants to know it - even those who do know it prefer something else (or so I'm told - I don't work here). C is more than 50 years old, but it works good enough for a lot of people and it is still getting better. C++ is nearly 50, and getting a lot of useful features. Rust is todays hotness, but it continue or just be another fad - ask me in 30 years.
I've seen many fads over my lifetime and I expect to see many more. Some fads I regret their death, while others I'm glad we saw the light and don't use that anymore.
Actually seen a ton of this from mid-career people who should know better but don't (or don't care). New grads aren't leading Greenfield multi-year projects.
Speaking as a mid-career who does know better, the thing is that I still have to compete in said competitive market along with everyone else and lots of recruiters/hiring managers/linkedin influencers/etc... don't seem to know better either.
I still remember the day I saw a posting for a job that required 10 years of Java, 5 Years of C#. At the time Java was about 8 years old, and C# about 3. I'll bet they hired someone anyway and nobody clued them into how impossible their requirement was.
Yeah, Resume-Driven Development is unfortunately such a common methodology and companies indirectly push for this by punishing candidates who admit to doing maintenance work and bug fixing instead of rearchitecting everything every year.
> We’ve seen (…) and would prefer (…) making money.
You think “the youths” don’t care about making money? That’s got to be in the top two reasons why people get into coding for at least the last decade. It’s also one of the top two reasons everything is shit, too many people only care about a quick buck.
Three times now I've argued strenuously against using MongoDB and Elasticache as the primary data store in an app, as they will inevitably turn into relational databases.
They did.
For Mongo going to postgres went great, but for complicated reasons we're stuck with Elasticache forever on our main product.
The only reason to ever use a non-relational db is for scalability and performance reasons. Joins and transactions are hard to do correctly and efficiently in a distributed system. So “NoSQL” solutions can be a good fit if your data is too big to fit on a single host and you can get by without joins and transactions.
(This is a massive oversimplification, but still used rule of thumb.)
And most companies vastly overestimate their data, and believe it to be "big", when it could be trivially handled decades ago by server-grade hardware.
> The only reason to ever use a non-relational db is for scalability and performance reasons.
And, most importantly, lack of market availability. Nobody is going to sell you a relational database nowadays, and rolling your own is... daring. Postgres was the last serious kick at a relational database, but even it eventually gave into becoming tablational when it abandoned QUEL and adopted SQL.
But tablational databases are arguably better for most use-cases, not to mention easier to understand for those who don't come from a math background. There is good reason tablational databases are the default choice.
Yes. The only wheel re-inventing I agree with is for learning purposes. I one would say rewrite SQLite, be my guest, it's a really good way to learn. But proposing it as a replacement is....bold. Might still be something to it, but that usually doesn't stem from a simple greenfield project.
It's often ignored because not all people get to work on interesting new features or care about the customers. This is all cool and dandy if you're a founder or working a job where the actual product is interesting, but most jobs are boring with no actual incentive or even a way how to care about the customer and then making it at least technically interesting in some aspect is a way not to go insane.
Just playing the devil's advocate here. I would prefer using boring technology that gets the boring work done as quickly and easily as possible anyways, but that's because I have more fun doing other things than working.
I think it is important to point out that boring tech is not Java 8 or .NET 2.0 that some companies are stuck with.
There is still lots of interesting stuff if your company uses newest Java or newest .NET but both are „boring” in a good way so mostly stable and changes to those are incremental and progress is stable.
Heck Angular with its latest improvements in v17 and v18 is quite interesting - but counts as totally boring and stable tech. Migration to signals and standalone components is a bit of a hassle but still is rather easy.
Hot new stuff is wish fulfillment by younger devs. They know it’ll take years to catch up with what the greybeards know about Linux or databases. If ever. If someone throws a curveball in then they reason everyone is on the same page. That nobody can be much more of an expert on 2 year old tech than they are.
What they don’t understand is that after 10 or 20 years of this drama you start to realize how much of this new shit is a repetition of the shit the “old shit” replaced, but with new jargon. Similar Shit, Different Day. Progress is not a ladder in IT, it’s a spiral. The scenery is never exactly the same but it swings through the old neighborhood all the fucking time.
At one point you need to admit Linux knowledge is unnecessary, beyond the basic cd/cp/chmox. Docker replaces it. Should I choose init.d or SystemD? None, just launch a new container.
I’ve spent too much time helping figure out why their container is broken to agree with this. It’s true that I use more general Unix knowledge than Linux knowledge, but that still counts for a lot.
I would only agree it can be a Bus Number skill rather than an everyone skill. My point is some day there will be a new OS the kids will embrace because everything is new to everyone and it’s their chance to shine.
When I worked at AWS as a support engineer I was unfortunately dumped into the containers team (EKS, ECS, Fargate, etc) despite being a "greybeard" in background.
A customer wrote in trying to figure out why his Fargate application kept crashing. The app would hit 100% CPU usage and then eventually start failing health checks before getting bounced (rebooted)
I relayed this back to the customer who insisted the app shouldn't be spiking in CPU usage and wanted to know why. Of course being a Fargate workload there's minimal ways to attach debugging to it. You can't just spawn htop on Fargate!
Doing due diligence I fired off an email to the team that managed the infrastructure. They curtly replied
"it failed healthchecks and got bounced"
"Okay but why"
"It hit 100% CPU"
"Okay but why?"
"It failed healthchecks and got bounced"
At no point were they either willing to interrogate or even consider the lower layers of the stack. The very existence of everything below the containerized app was seemingly irrelevant to them.
After going back and forth about this for nearly a month and a half with the customer I asked my boss to add me to the "Linux" support Slack channel. Reasoning that there's got to be other greybeards in there who (frankly) knew what they were doing better than these kids.
After writing a multi-paragraph explanation of all my findings along with the customer, moments before I hit SEND I got an email
The customers app was not releasing threads properly and causing the system to reach thread exhaustion and begin context switching, a CPU intensive process that eventually would take so long that the health check probes would breach timeout and say the app was down and restart it
Saying that "Linux knowledge is unnecessary" is to put it bluntly, ignorant to the point of clownishness. Having a holistic understanding of how a system operates is invaluable
> What they don’t understand is that after 10 or 20 years of this drama you start to realize how much of this new shit is a repetition of the shit the “old shit” replaced
That happens, sure, but all too often these same greybeards will fail to recognize when there is a truly novel thing. So do give new tech an honest chance before writing it off as reinventing the wheel.
Hyperscaler computer racks are reinvented supercomputers, and Cloud Software is quite frequently recreating features available on them in the 90's. Change My Mind.
> Because at the end of the day my customers don't know and don't care that I use Node vs Deno or Bun, or that I use NPM instead of pnpm, or that I'm not on the latest version of node or Postgres.
They don't know. And they shouldn't!
I don't have any idea what kind of tooling was used to create the drill I bought, just that it drills holes and is at the right price. Same for any other number of products and services.
The person who made the drill cares deeply, as they should. But consumers don't.
I'm starting to think there should be a term like the "HN effect" where engineers start to limit test into absurdities like "show HN: I built a rust backend with an elixir front-end that's actually running doom in pdf hosted on a distributed server composed of a network of the microchips in a Saudi camel herd, get fucked nerds" it sort of lends itself to this notion that the tools matter as much as the outcome (because oh boy do we love building tools, product be damned). I mean, building tools is fun compared to sitting down in front of a client and trying to explain to someone who can't use a mouse why they can't use the top of the line 4 gb machine from the 80s isn't working.
> absurdities like "show HN: I built a rust backend with an elixir front-end that's actually running doom in pdf hosted on a distributed server composed of a network of the microchips in a Saudi camel herd, get fucked nerds"
You leaned too far into the joke and it undermined the point. There’re roughly three kinds of Show HN:
1. “I made this product to solve this need I perceived. I hope it does well and makes me money. Come check it out.”
2. “I had this insane idea which is funny just from the description and made it as an experiment. Let me share what was interesting about it. Oh, and here’s the code.”
3. “I made this AI NFT Data Harvesting scam and think everyone is too stupid to notice this isn’t a real product filled with fake reviews. Come click this link from this account with 80% flagged comments.”
What you described seems closer to 2 than anything else. There’s nothing wrong with those. They stimulate curiosity, which is HN’s goal.
When I was younger and into electric guitar, I gushed over gear so much both at stores and on the internet that I forgot to actually practice playing my instruments.
In some cases people can even admit it. I know people who collect guitars. I know people who trade guitars. Both of the above can generally pick out a tune or two, but will admit they are not good.
I lust over a lot of gear (not guitar gear, but gear), but I keep telling myself that my goal is to play, and so I focus more on what I already have than the next gadget even though sometimes it really is my bad instrument that is the problem. Sadly, despite the above, I'm not any better than the other two groups (even when my equipment is a problem, better wouldn't help much since I'm still a large part of the problem)
Your drill was probably designed by an engineer under time and cost pressure, and then assembled in a factory by someone who absolutely did not care because they're thinking about going back to the sweatshop with slightly worse conditions but slightly higher pay.
No, but when your drill breaks there's still plenty of potential for it to physically injure you in different ways. As mooreds said, the end user doesn't care how you made the tool, they care that it won't go catastrophically wrong on them. How you accomplished that is not relevant.
Because software is living, often online, has various types of your and others' personal data, and usually runs on devices that you use for other things as well. The attack surface is massive, and more often than not, accessible. (Your drill might have a way to override the safety switch by soldering stuff to it, but you need physical access for that, and the reward is minimal; if you get into Equifax' publicly accessible servers, you get lots of important and lucrative data).
You might not care that your bank's website uses some obsolete Java framework with security holes, but you will care if they use it to drain your bank account. You might not care that your recipe generator app uses an old version of Jenkins for releases and doesn't do code signing, but you will if someone hacks them and releases a fake version that installs a keylogger on your machine, or exploits a bug in their backend to ship you malware that scrapes your Pictures folder with your nudes and important documents.
And on and on and on. A piece of software not updated for years, or a phone with no security updates, or with horrible legacy like Jenkins, is a security risk you as a user have to take at least some interest in. Yeah, you won't pick your bank by their tech stack, but when you see an atrocious website with UX out of the 1950s, absurd password length restrictions, broken domains/certificates/etc., you can be pretty sure their tech stack is a disaster, and maybe, you can try to avoid them.
I call it "Stackoverflowability". For every question you might have about mature tech, there'll be a list of similar relevant questions on SO that you can consult. The length of the list (and thus the probability of you finding your answer) closely correlates with the maturity of the tech.
I feel you on this, I've been planning out my next side project, and I've picked a Django based project that handles a good amount of what I need, I know I could build it all from scratch, but why would I invest a monument of hours for almost no benefit. I'd rather spend more time focusing on the key things that makes my idea special and not reinventing the wheel code wise.
being out of date on dependencies isn't the same thing as using mature tech. I sure hope you're updating on a regular schedule, for the bug and security fixes at the very least
Updates generate more bugs and more security problems than they fix, so that's not really an argument. If you've made your system sensibly, it will be immune to any security problems.
First, I want to see numbers to back that up. New versions bring new bugs that aren't known yet, by anyone. Old versions have older, more widely known bugs. From a stability POV, sticking with the old may be good: you know how to work around the old issues. From a security POV, that's probably bad: every script kiddie has a Burp Suite plugin to exploit it.
Second, there ain't no such thing as an immune system. You can asymptotically approach zero, like you can approach the speed of light, but it would require infinite resources to reach either.
Of course you can have an immune system, where things are not exposed or connected to where they could be vulnerable. For example, there is nothing that a "script kiddie" could write in the comment field of Hacker News that would be able to take control of your computer.
I don't buy into the "cyber security" arguments, and frankly I consider it a grift to keep hackers employed by playing on the fears of people. The same thing as "anti-virus" software, which never really worked in real life and isn't widely used anymore.
There have been image library exploits where uploading image to site that processes it gives access. The only solution was to update the library.
Or how about Heartbleed where the OpenSSL library had bug. OpenSSL is on the external web server and the attack could compromise server public keys. Perfect for impersonating the server. The solution was to update the OpenSSL library.
There have been browser zero days. Hacker News sanitizes input so user can’t compromise anything. But Hacker News could do an attack.
Consider a fairly normal web site that will send an e-mail from a customer form to the owner, with customer orders. That form is not connected to any private information or any money, at most you will get a spam order if the form is "hacked". Big deal.
Just between us, you do understand the point of an illustrative example, right? In this case, the person above me said you could have an immune system. I don't believe that's really true anymore. We've moved past it.
Keeping your dependencies up-to-date (at least updating known vulnerabilities) is very different than anti-virus software and the other check-list-oriented "security" industry.
The first is just blatantly irresponsible and dumb "advice", while I do agree that most of the "you need to tick this box in order to get the contract" kind of "security" software is just malware, and often worse than what they supposedly cure.
We're using a web host with an operating system and web server that are "obsolete" and haven't received any updates since a few years. There are no contact points where that server could access any of our machines. Not anymore likely than it accessing your machine. It serves hyper-fast web pages and receives customer orders. There's nothing sensitive there. If the server hall burned down or got hit by a tactical nuke, it would take 10 minutes to get stuff up on another server from backups.
For most businesses, credit card processing is outsourced to Stripe or similar services, and the security for that is their responsibility. Customer data is only stored on local machines with encryption. So it's very possible to architect solutions that aren't vulnerable. Unless you want to go into very unlikely scenarios.
In the worst case scenario, an attacker can send in one nonsense customer order that gets deleted by staff when they see it. This happens about twice per year. Customer orders are not stored anywhere on the server.
So you can't even fathom a scenario where an order is fulfilled without the payment going through, causing a huge amount of losses? Or leaking private data which is a huge deal in a post-GDPR world?
If you separate ordering, invoicing, and delivery, it is impossible for that to happen.
As for leaking private date, now you're in the territory of some hackers having access to reading RAM memory. Which I guess is a possibility, but not something that every business in the world needs to concern themselves with.
If you call your local auto dealer and say you want to buy all their cars, don't you think they have some process stopping them from just sending all their cars to your adress? A hacker could make that call, you know...
in my experience these new stacks, new frameworks, fancy new libraries, haven't added much in terms of user experience. I think React has but, with React comes a cacophony of ick.
Just this week, npm rotated some keys for the first time in a decade [0] and broke all of the sites I had deployed on multiple providers using pnpm. I had three sites on digital ocean which I had to switch to npm from to fix it.
The entire is ecosystem is the definition of not-boring
IMHO boring tech is great because it lets you focus on the actual tech of your product.
Well obviously you need to catch up with the times. If the 'CTO' (who has at least 5 years of experience) of my shiny new SaaS can't do a regular blog post about how clever he is solving this week's obtuse problem in version 0.3.75beta01 of this MemeDrvrUltra thing he found last month and bet the company on, is it even really a SaaS startup? And he'd be denied that multipart year end expose (crossposted to every social platform on the planet using AIoftheWeek v.五.九 to dress it up to fit) about HOW FUCKING HEROIC his team was staying up for 8 days straight migrating from MemeDrvrUltra to SuperMegaMemeblaster ("Closed Private Beta FTW! We're special") and how it almost worked because MemeDrvrUltra was at least a couple of years old and clearly not what the VCs were talking up at the last speed pitch angel event (tho version 0.4.22.beta03 did close a bunch of our tickets (what's EWONTFIX mean again?) and changed its mascot to some funny looking frog). If all you did in life is be old and lazy and boring and pick "what works" or "what's well supported" or "stuff that doesn't get me an outage call once a week at 2:45am", what's the point of even living? Really, what sort of loser wants to work at a company like that?
