> - "Add these domains with these mailboxes to an SMTP server running on my homelab, forward all incoming and outgoing via mxroute"
I laughed reading this. Have you actually tried it? I recently tried to get Claude (Opus) to give me a setup for a simple SMTP relay with Postfix. I had to constantly fight it not to include completely unnecessary components and configurations like Dovecot or just override my requirements. Even Claude is nothing but a semi-competent tutorial clicker.
To someone who never did anything like this themselves the result might seem fantastic. They might not battle Claude on details like I did because "hey, it works, doesn't it".
Don't get me wrong, I agree in general that LLMs are a significant milestone. But I would like to add, especially because of your tone, that perception of how much of a leap it is, is very likely a matter of individual competence. So your enthusiasm might say more about yourself than about the tool.
"To someone who never did anything like this themselves the result might seem fantastic ..."
I've been "doing this" for over 30 years now and I consider these LLM results to be fantastic.
If one has a deep, conceptual understanding of tools, systems and protocols and if one has the ability to carefully provide careful, concise specifications, these LLM tools are like being granted superpowers.
In fact ...
My next project upon which to bring LLM tools to bear is a fully local, solar-powered, self-hosted LLM toolchain.
Not only do I intend to never work without these tools again, I intend to do so without the ongoing environmental externalities that they produce.
> If one has a deep, conceptual understanding of tools, systems and protocols and if one has the ability to carefully provide careful,
concise specifications
Most people already don't have the ability to do this and their use of LLMs atrophies the little they had. "AI natives" will be completely doomed.
However, I never asked for LLMs nor did my consumer nor professional behavior send market signals that I would use them. They were thrust upon me apropos of nothing.
Now that I find them useful, the least I can do is stop making the problem worse.
As a rule of thumb the energy use to produce a silicon wafer is roughly equal to the energy it will use in its lifetime. For GPU and VRAM this is considerable. To equip yourself with the capability to run an LLM locally is making an upfront investment in the production of extremely toxic and resource intensive products.
Why buy a personal EV with 600 mile range that sits idle 95% of the time when you can take a bus (aka the cloud) that maximizes the resource utilization?
One of the most wild parts of the "Is AI useful" arguments is how different people are getting so astoundingly different results from it. One guy will claim that he can one-shot "Port this unmaintained app from 2017 to modern Android," go do the dishes, and come back to a working application. And the next guy will say, "I can't get it to do anything groundbreaking and have to micro-babysit it every step of the way to the point where it is more work." Assuming both people are truthful and operating in good faith, how can this be explained? How can simply writing a prompt in English be a "skill issue?"
I'm personally much more towards the "I struggle to get it to work well" side, but there have been so many anecdotes from people who are earnestly saying it does all their work for them and that they hit enter, go to bed, and wake up to working software. I'm trying to imagine a hammer or band saw that produces such wildly different results in the hands of different users.
I'm the GP that got Claude to port the app, set up the SMTP server etc. etc.
And I do see stuff on that level of complexity, multiple times a week. I'm not a full-time software dev any more, but did learn the craft at a really good company before AI came along - and I've been doing basic sysadmin stuff by hand, almost exclusively on the terminal, for almost a decade now.
A lot of pithy responses to your comments that contain some pretty bad-faith assumptions about AI "power users", but to answer your question I think part of it genuinely does come down to a "skill issue" - or at least familiarity.
AIs really do have huge blind spots, and there are things it can't do well. But on the flip side, there are modes of operation that do get good results and if you keep it in this "zone" you can get amazing results.
Note the "Working with unfamiliar data or systems" section - it doesn't stop the models from making wrong assumptions but it does get them to test these assumptions with simple experiments and course-correct before human eyes ever see the result.
Setting up things like CLAUDE.md files and skills so that it doesn't make the same mistakes over and over is a big help too. The model doesn't literally get smarter, but it knows how to avoid pitfalls its fallen into in the past.
If you want to have a chat about it earnestly, my email's in my profile.
My guess is they do something like "Hey dingus, make my app send emails" and then Claude or GPT just goes off and does a thing that makes it look superficially like it sends emails but it really only sends email exactly on Tuesdays and if the subject line is "send email to user". And there's no tests, so who knows if the next LLM can fix the bugs without introducing new ones.
For the last several years (since GPT 4o) I have written several apps in use by people, making money, entirely with AI/LLMs (me neither writing nor reviewing the code in any meaningful fashion - other than high level architecture, schemas, etc.) and - yes, in a few hours it can do things that would take a normal human weeks (if ever!). But left unconstrained, it will just pile more and more garbage on the pile.
Fable MAX is the thing that gets me much closer to "just send emails" but even then it doesn't look in my specs directory for the spec, it just goes off and does a Jurassic Park-style "It's a Ruby-on-Rails system! I know this!" and disregards all the other ways we send email and writes its own thing. And often I'll go "Where's the button to do X" and it will say "You're right! Nobody asked for it, so this page is an orphan!"
I happen to use Superpower's (https://github.com/obra/superpowers) "brainstorm -> spec -> plan" workflow in e.g. Fable MAX (for anything non-trivial) then I have Fable send to GPT 5.6 Sol XHIGH for execution, with Fable (either the original, or a separate one, depending on the criticality of the task and the blast radius) being the critical reviewer.
Even then, though, I need to continually guide it against the norms and conventions of the codebase/app, because it makes a ton of assumptions.
It doesn't surprise me that if people don't take a fairly rigorous approach to AI-software development then they'll end up with a mess. Even if you buy into "just re-write it" (I happen to think that's where we're ending up) - we aren't there yet and without e.g. a strong test suite, re-writing it is just as likely to create more bugs than it is to fix the existing ones.
that's a great point. i personally am toward the "i go to sleep and it does my work for me" side, but share the same confusion about how we went from 100x-0.1x programmers to 100000x-0.0001x ... vibe coders. to be clear: i am aware of what my side of the fence vs the other looks like, and i don't find anything about "meh iono it doesn't work that great for me" unreasonable at all.
> Assuming both people are truthful and operating in good faith, how can this be explained?
Quite simply, difference in skill.
The sysadmin that has configured a few applications in their career would likely get annoyed at how many mistakes an LLM makes.
The manager who barely knows their way around a shell, uses an LLM to configure an application in less than the three business days it will take them, and will claim they are 10x more productive now.
Once again, LLMs are effectively great at triggering the Dunning-Kruger effect. The least you know, the more productive you feel.
Here's the thing. I've been an AI denier and skeptic since the beginning. I've softened. This is my reality on the ground:
I am tech lead on a front end project that is typescript with react and everything else as bog standard as possible. In the last year the backend devs have been pressured to be full stack, so have been submitting AI PRs to my repo with Claude. The PRs mostly 'work', but invariably have major problems that wouldn't make it through a code review in the before times.
Front end JS on a CRUD app is the literal best scenario for AI. Professional senior devs are using it, and it can kind of work, but still not really.
I've given it the old college try myself recently, and it is at the same time astonishing and disappointing. Yeah it can do a lot, but there's always something wrong with it. The 'something' is not always a bug; it can be a design issue, performance issue, unused code, duplicate code, etc, thus many people don't notice the problems. But I think for those of us who prided themselves on quality, the stuff doesn't work well enough to use. For those who leaned more towards speed, they're overawed. I am concerned for my career, yes.
that’s perhaps a bit harsh, and I’m late (intentionally) to all this as I’ve worried about personal atrophy, security, and feeding the shoggoth
but also with my instructions it will churn on prompts for hours. It makes use of my local rags by domain (which it also helped me build and maintain). Most recent
example: it’s been on my list to set up headscale for like 5 years now. I know how to do it, I’ve read the docs and kept putting it off, partially cuz Tailscale works and has a generous free tier, and partially because I didn’t want to strand assets that I’d somewhat shortsightedly hardened to only accept ssh from my Tailnet.
claude did it all in 3 hours, while I worked on other stuff, prompting for `op` access when it needed keys, and logging each usage, with minimal involvement from me. All with more documentation, roll back points, and “learnings” propagated to the appropriate rags than I could have created in a weekend session. It made mistakes (some that I likely would have made, like region settings) then reasoned about it, corrected and tested, all transparently, all git backed, all posted about on my Mattermost server as it went, so I can easily search it the way I would of an extremely detailed teammate did it.
I don’t know what to make of it all, but letting it churn through my backlog has been eye-opening.
surely this is closer to the common experience with LLMs... i don't know how people don't see it for the miracle it is. it's like if God himself came down from the heavens and said "no one will ever die of hunger ever again, but you have to run 5 miles a week" and we all got bogged down in how hard it is to run 5 miles...
"it must be some sort of weird badge of honor to be so profoundly incommunicative that you can't get the ai that desperately wants to sycophantically pump you up about all your brilliant ideas to be that useless. that is genuinely harder to do than simply using the technology even passably."
Definitely incomprehensible that someone might have a different take than you about the 'goodness' of your prompt outcome...
ah, so the soft implication that you are simply smarter than me. fine. there's at least a 1/100,000 chance of that. no one forced you to be that smug and dismissive, though-- seems like a few traits that are fairly antithetical to learning. but i guess you don't need to learn, since you have already ascended to Enlightenment.
I mostly view AI as raising me up one more level of abstraction in coding. It still requires babysitting and verification if you want anything vaguely maintainable that's not pulling in a ton of irrelevant libraries, but now I am able to give a clear spec for a ~50-100 line task and it'll get it 90% right, and it's a small enough chunk I can validate it well.
Claude though is still slow enough that I think it works out to maybe a 50% improvement at my time to do a task, but sacrificing deep knowledge and learning. I worry that I'll never develop the same skills and intuition that senior engineers developed a decade ago because I'm outsourcing a lot of the planning and thinking.
I laughed reading this. Have you actually tried it? I recently tried to get Claude (Opus) to give me a setup for a simple SMTP relay with Postfix. I had to constantly fight it not to include completely unnecessary components and configurations like Dovecot or just override my requirements. Even Claude is nothing but a semi-competent tutorial clicker.
To someone who never did anything like this themselves the result might seem fantastic. They might not battle Claude on details like I did because "hey, it works, doesn't it".
Don't get me wrong, I agree in general that LLMs are a significant milestone. But I would like to add, especially because of your tone, that perception of how much of a leap it is, is very likely a matter of individual competence. So your enthusiasm might say more about yourself than about the tool.