A local model needs 0 investment and 0 commitment, takes literal minutes to get started (especially if you have someone who is into that stuff showing you the ropes) and if you end up disliking the experience of using AI you can just `rm -fr` it and forget the whole thing existed.
- You at least need a capable machine, so that's not 0 monetary investment.
- You need to spend at least an hour decicding between ollama, llamacp, mlx, etc.
- You need to find the correct quantized version of the model that works for you based on the architecture.
- You need to figure out the correct context window size to get reasonable performance.
- You need to setup a harness that works against your model
- You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI.
Or, you could sign up for Opencode for $10 and just be productive.
I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
I'm running Qwen 27B no problem with an AMD 9070XT + 24gb DDR5 ram. Does basic web search for me (tool call with tavily, costs nothing I get 1000 searches a month) and is great for creative writing (primarily breaking writer's block). Until the recent surge in ram costs, that wouldn't be hard to do. I built the computer for ~$1600 a year ago.
I am getting ~13-15 tps with my 9070XT for the 27B (~35tps for the 35B-A3B), but I think for me the main bottleneck is the 64gb of DDR4 3600 memory. What kinda speeds are you getting with what speed of DDR5?
I’m a little more novice than a lot of the people on this site so take my response with a grain of salt.
The wall I keep hitting is I can run models like I described (Q3-4 usually), but it’s very sensitive to context. Once I start getting past 7.5k or so it can really fall apart. Sometimes before that. It just really depends.
If I run smaller ones that offload less to ram, they stay somewhat coherent but don’t quite do what I want them to do.
Your token speeds are not that much slower than mine. I imagine part of it is I’m not fine-tuning it very well. On a good day I’ll get like…15tps.
Back in “the day” nerds just bought the hardware to fuck with. Some of us still do. Claiming that compute is the barrier to entry just means you’re not a nerd. That’s ok.
> You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model
How is that hypocrisy? Self hosting is somehow anti AI? Its not anti AI. Its literally using AI!
…and honestly, at a higher technical level than slapping your wallet against a token provider and running prompts in a hosted sandbox you can't even see the prompts in.
All of your objections have already been addressed by the previous comments.
The original comment states that the person in question already had a suitable graphics card to hand, so it did not require a monetary investment.
GP clearly states that "someone who is into that that stuff" was guiding the process, so it did not require a significant time investment.
> I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
I see no hypocrisy in the original comment.
You've also assumed the skeptic in question doubts the capabilities of AI. That may be the case (like you, I have no idea), but they may also have privacy concerns, in which case a local model is the appropriate choice.
There are plenty of reasons to be skeptical of AI.
install LM Studio, download the automatically selected quant based on your hardware, start a conversation with the automatic context size. 10 minutes at best and zero effort
Look man I am incredibly skeptical of how LLM’s have been rolled out and all the promises people make (it’s so much snake oil and pipedreams), but I also found it very trivial to hop on LM studio and start tinkering with models. If you’ve already got a decent midtier computer on hand, which I imagine a lot of us already do, then it’s really not hard to get started and get immediate results.
Maybe not '0 investment and 0 commitment', but incredibly little depending on what you have laying around. It takes less than 5 minutes to download say LM Studio and an Open Model and as long as you have the hardware to support it, you start moving along. If you are on AMD in some ways it's even 'easier', you can download Lemonade and it will tell you exactly what will fit and best options based on what you are trying to do.
For me at least the local AI stuff, powered with solar has been pretty great. Would that scale to a large business? Goodness no, but for my tinkering and learning, it works great.
This is the diametric opposite of the rent-vs-buy scenario that this entails.
Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware.
Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine.
> takes literal minutes to get started
Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers.
Vendor: Free-tier available instantly on a web URL. Even local agents have free tiers from multiple vendors. Install is a single command and/or download and "next,next,next,finish" wizard that takes ~1 minute.
> you can just `rm -fr` it and forget the whole thing existed.
I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Meanwhile I simply... stopped using Gemini. That was the entire process: I no longer actively use it. They stopped billing me for my token usage, because it is now zero. That's... it.
> I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Are you trying to say that local models are hard to use because... you're having issues handling files properly? I am not sure I get the argument.
I get the rest of the comment: local models require an investment upfront, and it is less convenient. It doesn't say that it is not cheaper, though.
It took me about three hours total to set up a local model. I already have a GPU and I have fiber for the download. llama.cpp is not difficult to compile and has many backends. It can run parts of the model on different backends, like in the common case that the GPU doesn't have enough VRAM for everything. There are many step-by-step guides available.
Takes even less depending on your system. LM Studio or Lemonade and you are set up in minutes and now they can even tell you what models will fit with the memory you have.
And it would be in seconds if models weren’t that large and slow-ish to download! LM studio is such a noob friendly experience, pretty neat first experience!
At least for the most part, if you are downloading from huggingface, you should be able to saturate your connection. I know I usually can pretty easily even with a 5gig connection at home.
PS: This whole thread reminded me of several managers I've worked with who were pathologically unable to estimate... anything, be it driving time or development effort.
They always focused on the "minimal aspect", ignoring everything before and after. Walking to the car park. Standing in line at the machine. Paying at the machine. Getting out of the car park in the car, surprisingly long during busy times. Driving through traffic. Any delays that could -- and regularly do -- occur. Finding parking. Actually parking. Walking from the car park. Etc.
> "I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!"
I used to deal with these kinds of frustrations too.
fd --unrestricted --size +1G
fd --help
-u, --unrestricted...
Perform an unrestricted search, including ignored and hidden files. This is an alias for
'--no-ignore --hidden'.
-S, --size size
Limit results based on the size of files using the format
<+-><NUM><UNIT>
Full model or a 4-bit quant? I have a 5090 and I'm not sure whether I should use a quant that fits within the VRAM or a much bigger version where I'd have to offload a lot to 64GB RAM and a beefy CPU (but still a CPU)
I personally run the Q6 quant on my RX 9070 XT (16GB VRAM). On r/LocalLlama there was a post recently as well, which talked about the degradation of different quants (for the 27B version)[0]
A lot of people already have 12GB+ GPUs lying around for playing games, doing video editing, etc. I would not get a GPU or mac just to run LLMs personally, but if one wants to get such a device for other tasks too, it may make sense to eg choose a slightly higher (v)RAM variant if they want to run some bigger models. Then what you pay for the local llms is just the difference.
“If you have the prerequisite hardware, then… know which model you want out of thousands of a variants… and your drivers are up to date, then it is fast!”
Literally tens of millions of people of silicon MacBooks have sold send 2020 so it’s probably safe to say hundreds of millions of people have the necessary hardware. Not even getting into smartphones.
>which model you want
Have you personally searched for models in LM studio? It’s actually pretty straight forward and it tells you with a very clear icon if it will all fit in your GPU or if it will offload onto ram.
>drivers are up to date
Are you just making things up now? I run LM studio on an M1 MBpro (albeit very small model for small tasks with tool calls) and on a Linux (fedora) PC with an AMD GPU. In both cases i downloaded LM studio, quickly found models with their search, and started messing around. I am not a coder or engineer mind you, so clearly it isn’t that difficult.
Local models on regular hardware aren't really capable of anything. Whatever you're testing is nowhere near a measly $20/mo subscription, so it's of limited use.
Your statement comes in extreme contrast with my experience using local LLMs for more than year now. Qwen 35B-A3B, even its Q4 quantization, is extremely capable. Hell, at this point i pretty much turn on my PC, then run llama-cpp just to have it in the background for when i need it to do stuff.
My local model (Qwen 3.6 27b) with a basic web search tool call has proven a fantastic “dumb” assistant editor/writer when writing. It’s really breaks through my writers block.
It’s limited, it’s a bit finicky here and there, but it’s been very helpful. Mostly just gives me nuggets of ideas or shows me what I don’t want, which is informative.
i really like the idea of running local models but i'm always in the position of wanting the best model(s) available and i don't have any severe privacy concerns. as such i have yet to justify ever using local models.
