With almost 600 books in my kindle collection over a period of about 15 years, I would like to think I was a relatively active customer. When they announced “your kindle books are just a license to read”, which happened about the time they announced the deprecation of the old format, I went and converted the entirety of my library to Calibre with multiple open formats.
That was in December, I have not bought a single book on Amazon since then, and the kindle app is not installed on my new phone. Just in case anyone from the relevant AMZN department is reading this.
Architecturally - where do you run Postgres ? I assume it would be external to the cluster ? (doing it internally would create a circular dependency ?)
I’ve made something very similar that is almost backend-agnostic: https://github.com/ayourtch-llm/tttt - and it does auto inject the MCP in case of Claude, but it is trivial to adapt to other backends.
I'm a (non-practicing) Dwaitin Hindu. AFAICT, there's no mainstream school of Hindu philosophy (there are three) espouses that view. Although, Advaitins come very close to it with their four mahavakyas.
IMO, Integrated Information theory of consciousness (IIT) is exactly that. Everything is conscious, the difference is only in the degree to which they are conscious.
I tried qwen3.5:4b in ollama on my 4 year old Mac M1 with my own coding harness and it exhibited pretty decent tool calling, but it is a bit slow and seemed a little confused with the more complex tasks (also, I have it code rust, that might add complexity). The task was “find the debug that does X and make it conditional based on the whichever variable is controlled by the CLI ‘/debug foo’” - I didn’t do much with it after that.
It may be interesting to try a 6bit quant of qwen3.5-35b-a3b - I had pretty good results with it running it on a single 4090 - for obvious reasons I didn’t try it on the old mac.
I am using 8bit quant of qwen3.5-27b as more or less the main engine for the past ~week and am quite happy with it - but that requires more memory/gpu power.
What matters for Qwen models, and most/all local MoE models (ie. where the performance is limited) is memory bandwidth. This goes for small models too. Here's the top Apple chips by memory bandwidth (and to steal from clickbait: Apple definitely does not want you to think too closely about this):
M3 Ultra — 819 GB/s
M2 Ultra — 800 GB/s
M1 Ultra — 800 GB/s
M5 Max (40-core GPU) — 610 GB/s
M4 Max (16-core CPU / 40-core GPU) — 546 GB/s
M4 Max (14-core CPU / 32-core GPU) — 410 GB/s
M2 Max — 400 GB/s
M3 Max (16-core CPU / 40-core GPU) — 400 GB/s
M1 Max — 400 GB/s
Or, just counting portable/macbook chips: M5 max (top model, 64/128G) M4 max (top model, 64/128G), M1 max (64G). Everything else is slower for local LLM inference.
TLDR: An M1 max chip is faster than all M5 chips, with the sole exception of the 40-GPU-core M5 max, the top model, only available in 64 and 128G versions. An M5 pro, any M5 pro (or any M* pro, or M3/M2 max chip) will be slower than an M1 max on LLM inference, and any Ultra chip, even the M1 Ultra, will be faster than any max chip, including the M5 max (though you may want the M2 ultra for bfloat16 support, maybe. It doesn't matter much for quantized models)
But it doesn't matter if you have 1000GB/s memory bandwidth if you only have 32GB of vram. Well, maybe for some applications it works out (image generation?), but its not seriously competing with an ultra with 128 GB of unified memory or even a max with 64 GB if unified memory.
> but its not seriously competing with an ultra with 128 GB of unified memory or even a max with 64 GB if unified memory.
No one is arguing that either, this sub-thread is quite literally about the memory bandwidth. Of course there are more things to care about in real-life applications of all this stuff, again, no one is claiming otherwise. My reply was adding additional context to the "What matters [...] is memory bandwidth" parent comment, nothing more, hence the added context of what other consumer hardware does in memory bandwidth.
If we are talking about Apple silicon, where we can configure the memory separately from the bandwidth (and the memory costs the same for each processor), we can say something like "its all about bandwidth". If we switch to GPUs where that is no longer true, NVIDIA won't let you buy an 5090 with more 32GB of VRAM, then...we aren't comparing apples to apples anymore.
There is also prompt processing that's compute-bound, and for agentic workflows it can matter more than tg, especially if the model is not of "thinking" type.
I am 50, coding since ~12. Started with Apple II, during the uni times wrote my own editor in assembly for BK-0010 (a soviet computer), then 30 years in computer networking with some high performance dataplane stuff more recently;
The last years somehow it felt like there’s nothing new anymore, the same 10 ideas being regurgitated with slight modifications. I tinkered with AI for the past 2 years but it was mostly a “tool for writing boilerplate”. I have tried a few ideas for agents but didn’t see how it could work.
That changed with Opus 4.6 and the subsequent wave of local models - now I try 10 ideas a day and it’s like magic! And if something doesn’t work - jumping into the code and debugging it is huge fun!
Understanding that the era of the almost-free cloud tokens might come to an end, I run my own harness pointing to my own GPUs running Qwen3.5-27B, and the last few days it has been very busy! :)
My harness doesn’t “pressure cook” since it doesn’t make sense to do that with only one GPU (besides many other reasons), it runs everything in a linear fashion, including subagents, and logs everything - reading the logs as they go by is another cool thing - sometimes I pick up interesting things from it !
The distribution of people’s moods related to AI seems indeed bimodal. And I feel lucky somehow ending up in the “enthusiastic” rather than “depressed” part of it. To the folks in the other one: I am sorry. I don’t know why it is this way. If I knew I might have given unsolicited advice.
So you’ve tried at least a hundred ideas by now, care to share fifty of them? I’m very curious as to what they are. Opus is too slow to even complete one idea per day for me, and that’s fine, I don’t have hundreds of them :)
I dont have big ideas. Some of the more interesting ones that I ended up using but can’t share: a streaming radio for my MP3 collection (runs behind the vpn); a lightweight and self contained webrtc conference server for talking with my family; a process-level virtualization based on KVM.
Also LLM helped with a lot of code for my packet mangling library: https://github.com/ayourtch/oside - which, among other things, includes a now battle tested SNMPv3 stack.
These are the ones I remember. Feel free to scout my GitHub for more. Edit: And of course it doesn’t need to be said that out of ideas I try all of them make it to github. Many end up thrown away.
That was in December, I have not bought a single book on Amazon since then, and the kindle app is not installed on my new phone. Just in case anyone from the relevant AMZN department is reading this.