I hope I'm not the only one who misread it as 292 tokens/s and got excited momentarily
361994752 7 hours ago [-]
I thought I met some huge breakthrough on computer science until I saw your comment
awesomeusername 9 hours ago [-]
Most definitely not.
Weeps...
senectus1 7 hours ago [-]
haha its something i look out for now. having seen a few recent stories like this.
bensyverson 17 hours ago [-]
I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?
pizza234 17 hours ago [-]
> but are they functionally useful or helpful to anyone currently?
Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.
MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.
The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
ddevnyc 16 hours ago [-]
Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
dannyw 16 hours ago [-]
MoE has nothing to do with multimodality.
MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.
Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
> There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
Huh, always thought one would sort of require the other.
For a good MoE model, wouldn't I want to minimize the "churn" between experts, i.e. the amount of time one expert model has to be swapped in for another expert model? That would be naturally the way if experts correspond to semantic categories.
E.g. suppose I have a model that can answer questions in 100s of languages. Then while any of those languages might be requested by some caller, it's highly unlikely a caller will request all languages in the same session - realistically, there might be one or two languages in a session and those will then span the entire session.
There will also be languages that are requested very often and others that are extremely rare.
(Let's say my model also supports Klingon and Sindarin. Those are important for marketing reasons and because I genuinely like to make the occasional nerd happy - but practically, I get maybe a handful of requests for those every few months. So it would make sense to centralize the knowledge for those languages in some specific part of the model, so I can keep that part out of VRAM - and probably RAM as well - during the 99% of time where it's not needed)
So wouldn't it make sense to make the expert models language specific here? Then you could take advantage of the fact that a language rarely changes inside a session and keep that expert in VRAM for the entire session. You could also avoid dragging parameters along with you for languages that are practically never used.
dummydummy1234 14 hours ago [-]
Wait I thought the router ends up specializing the experts?
Like there is no explicit goal aside from each 'expert' getting roughly equal weight?
And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?
I do not know what I am talking about, this is my limited understanding...
ahepp 9 hours ago [-]
My understanding of it is also pretty surface level, but I was not under the impression that it develops "expertise" in a particular subject matter, at least not in a way that's easy to harness. From what I've read, it develops expertise at the token level.
Because the natural continuation of this is to say like, "Ok I want to load the bird detection expert and the navigation expert but leave the medieval European history expert behind", and my understanding is that this is not really how it works. At least at the moment.
juliendorra 9 hours ago [-]
You are right that at training the main goal is balancing between the sections to avoid certain paths becoming the only path. In the end the inference will be routed token by token to a mixture of say 3 or 4 sections of the model. The combination can change at each turn. It’s really a statistical optimization.
As for many things in neural networks, the original intuition coming from anthropomorphism once implemented becomes something very non human!
ahepp 11 hours ago [-]
> You can think of it like sharing.
was this meant to read "sharding"?
bensyverson 16 hours ago [-]
Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.
reactordev 16 hours ago [-]
yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.
bensyverson 16 hours ago [-]
And is this using conventional model loading (all in VRAM), or are you streaming it in some way?
cyanydeez 12 hours ago [-]
in theory, QWEN3.6-35B-A3B should run as fast as a 3B model; and in practice, it can be just as dumb.
dsl 11 hours ago [-]
> but are they functionally useful or helpful to anyone currently?
I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.
14 hours ago [-]
nickpsecurity 16 hours ago [-]
If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote.
If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful.
If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.
parineum 15 hours ago [-]
at 5 minutes per token, you could look at the results next week
bensyverson 14 hours ago [-]
one week later: "It says 'You're absolutely right! Let me look at the seams so I'm checking, not guessing—' and I guess that's when my SSD melted."
hex4def6 11 hours ago [-]
Turns out the SSD was load-bearing.
nickpsecurity 10 hours ago [-]
That's way slower than I thought it would be. I struggle to imagine a use case.
If you had a deadly condition, and no diagnosis worked, and a specific model had the answer... past that I wouldn't use it.
zapataband1 11 hours ago [-]
I don't need any of these to help with coding. skill issue.
xlayn 5 hours ago [-]
I have see a lot of "run <gigantic llm model name here> using cpu" or using 1gb of ram... and the short answer is sure, you can run it in 1mb of ram, or in a 286, it will just take a couple of years to produce the first token.
the canary in the coal that this has not change is nvidia share price.
xg15 12 hours ago [-]
At that point, how does this compare with simply running the model on the CPU?
sixothree 8 hours ago [-]
Not an answer to your question, but maybe this has more info? I think with these optimizations and quants, it compares negatively. But can these optimizations be applied to models you want? Another question.
https://news.ycombinator.com/item?id=48353348
jackb4040 18 hours ago [-]
Ahaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!
throwawayffffas 17 hours ago [-]
Running 292 tps on K3 would make your gpu a money printer.
meneton 15 hours ago [-]
spt not tps
root-parent 13 hours ago [-]
It matches my coding speed...its ok.
killingtime74 7 hours ago [-]
it takes you almost 5 mins to write at most 1 word?
weberer 6 hours ago [-]
When averaged over 24 hours, probably.
xnorswap 17 hours ago [-]
I wonder what this measures in J/token.
throwawayffffas 17 hours ago [-]
Assuming 30% gpu power utilization because of all the loading and unloading 29.2 kJ per token
donquichotte 16 hours ago [-]
Damn 15 AK47 bullets per token
mring33621 17 hours ago [-]
How many is that in tokens per Scaramucci?
fhars 12 hours ago [-]
It is a bit more than four tokens per millifortnight.
logicallee 17 hours ago [-]
that's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device.
For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.
So what you get is 80x more expensive and you wait 416 days to get it.
17 hours ago [-]
aaroninsf 14 hours ago [-]
Wow, you could do a lot at 292 tokens a sec—oh.
I have all praise for those taking this on and in my idiom would call it *the lord's work."
The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.
It's work like this that moves things forward.
exe34 15 hours ago [-]
Hah I was looking for it and couldn't work out how many years/token. 292s is pretty good.
roger_ 18 hours ago [-]
Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained.
Hoping a winner emerges with some real momentum behind it.
akie 17 hours ago [-]
No no, don't just say "vibe coded", say "Fable and $500 of credits"
ricardobayes 12 hours ago [-]
Or Deepseek with a fiver.
Foobar8568 16 hours ago [-]
Opus 4.6 was already enough to tackle these projects vibe coding.
gwerbin 16 hours ago [-]
Incidentally I think I was more productive with Opus 4.6 than with any subsequent Anthropic model.
cute_boi 11 hours ago [-]
Yea, opus 4.6 is much better than opus 5 because it is willing to do task. Opus 5 just gaslights and refuses to solve many complex problem.
Lalabadie 17 hours ago [-]
"This one is made with premium vibes"
cyanydeez 12 hours ago [-]
these things can be done with local models; so don't say 500$ credits, say "accelerated climate change disasters by $500/year"
hiramwen 18 hours ago [-]
You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.
tecleandor 15 hours ago [-]
So, instead of having one or two maintained projects that work super well, now we have hundreds or thousands of people half-assing it locally each time they need it? Doesn't sound efficient.
serf 10 hours ago [-]
you're right, but flipped on its' head it also means that a well-meaning FOSS project may not ship out a zero-day in the future that affects half the planet at once.
now we'll just always be vulnerable from different angles in different and unique ways across the globe.
better? well, no.. but it is different, and biology has offered tons of wisdom about why it's a good thing to achieve things different ways across the world, maybe some far-future version of this weird fragmentation of work that is happening now will contribute to some kind of herd effect that reduces the severity or magnitude of some bug or work of malice.
who knows though, i'm probably just considering it too positively. The disintegration of consolidated effort is a pita, I agree.
tecleandor 8 hours ago [-]
There has to be a middle ground between a monoculture and not sharing a single knowledge bit between every human. Because if everyone is making their half assed thing just for themselves using LLM without sharing anything, that also means FOSS will stop existing, and knowledge will only evolve inside the corporations that have money for creating their provate knowledge bases.
IAmGraydon 13 hours ago [-]
It might make sense to do this if there literally isn't an existing solution out there, but I think most would find that it's better to use an existing solution than to recreate the wheel every time there's a need. It's just less effort/time/money.
esafak 14 hours ago [-]
I have the same objection to forking open source projects; changes should be upstreamed when possible.
titularcomment 17 hours ago [-]
This is complete dependency on LLM tools all the way from development to usage, and is risky as well as prone to failure
hiramwen 16 hours ago [-]
Local inference without access to internet and tools is pretty safe, worst you get is a slop output, best case it solves your prompt.
mkagenius 11 hours ago [-]
I love how goal posts are shifting from "vibe coded apps dont really work" to "vide coded projects wont be maintained"
classified 17 hours ago [-]
How a patchwork of Python modules constitutes an API(?) or whatever(?) and how to use it (does anyone?), is beyond me. There is nothing that I would call documentation, let alone concrete usage examples. I know more amusing ways to waste my time. When I want to play with models, I use llama.cpp.
hatthew 8 hours ago [-]
I've been seeing a lot of projects that let one use large models on machines with small amounts of memory. They seem to all be doing significant quantization and/or expert streaming. What's the benefit of these projects over something like taking an unsloth quant and running llama.cpp with appropriate flags (-cmoe/-mmap) to manage VRAM vs RAM vs SSD?
seu 13 hours ago [-]
I love it how the rampocalypse is pushing people to squash all the performance they can. I hope that this also leads to rethinking model architecture so that they can do the same with less.
cpfohl 18 hours ago [-]
I’m still slightly confused on what this adds.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
dofm 18 hours ago [-]
If you point it at a huggingface model identifier, it will download it, I assume. No way around that.
It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
pvtmert 17 hours ago [-]
Not an expert in this field, but the "expert" is consisting of multiple layers. To keep it small in terms of memory print, this project streams each layer (dividing even further).
It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity.
Also guessing that you need to be able to hold at least 3-layers at once in the memory, given M x N = R operation, M is the previous layer, N is next, and R is the result. on the next "layer", the R (result) becomes M, gets computed against the next layer, N, yielding the further result R'. And so on, until all layers are processed.
I assume it's horribly slow, but can be put in a non-intrusive background task...
dofm 11 hours ago [-]
Thanks for the extra detail here.
cpfohl 17 hours ago [-]
Obviously needs downloading eventually :).
It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.
jedbrooke 15 hours ago [-]
I mean, if you had fast enough internet connection, you could just stream it over https
junsu22 3 hours ago [-]
"The layer-by-layer streaming is clever.
Curious how the throughput compares to running a quantized model on the same
GPU — seems like quantization might still win on speed?"
xg15 12 hours ago [-]
You can run any frontier model on your PC if you just wait long enough...
ilaksh 18 hours ago [-]
I guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?
speedgoose 18 hours ago [-]
And you don’t pay for the electricity.
mrwaip 14 hours ago [-]
What device do I need and how much will it cost to install one at home so that it works as quickly as the Claude Code answer (and it answers quite slowly)?
SwellJoe 7 hours ago [-]
If you want it to respond in under a minute, you need more hardware than this application is intended for. This thing's response is measured in seconds per token, not tokens per second.
To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which is not fast enough for comfortable interactive agentic use. A couple of new GPUs, like Radeon AI Pro 9700 at $1400 each, probably gets you fast enough for comfortable interactive use with small models like those. Those small models are not competitive with Claude models, however (maybe they beat Haiku sometimes). They can write a little Python or make a web page, they can't architect a real application.
To run a near-frontier model like Kimi K3 or GLM 5.2 at comfortable speeds, you need serious hardware with 768GB VRAM, minimum. I think Asus is releasing something like that for about $150k soon. You can run DeepSeek V4 Flash at almost comfortable speeds and in a decently capable quantization on two DGX Sparks or Asus GX10s (about $10,000).
Or, you could use DeepSeek V4 Flash from DeepSeek.com, at blistering speeds and with huge contexts, for something like a decade or two for that same $10,000.
book_mike 17 hours ago [-]
We will see if this project has legs. This is the kind of efficiency we desperately need. Now if we can address efficiency with llm training.
myshapeprotocol 17 hours ago [-]
Running 70B on a 4GB GPU is wild. Really impressive engineering feat for resource-constrained environments.
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
https://github.com/lyogavin/airllm/releases/tag/v3.1.0
Weeps...
Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.
MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.
The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.
Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171
Huh, always thought one would sort of require the other.
For a good MoE model, wouldn't I want to minimize the "churn" between experts, i.e. the amount of time one expert model has to be swapped in for another expert model? That would be naturally the way if experts correspond to semantic categories.
E.g. suppose I have a model that can answer questions in 100s of languages. Then while any of those languages might be requested by some caller, it's highly unlikely a caller will request all languages in the same session - realistically, there might be one or two languages in a session and those will then span the entire session.
There will also be languages that are requested very often and others that are extremely rare.
(Let's say my model also supports Klingon and Sindarin. Those are important for marketing reasons and because I genuinely like to make the occasional nerd happy - but practically, I get maybe a handful of requests for those every few months. So it would make sense to centralize the knowledge for those languages in some specific part of the model, so I can keep that part out of VRAM - and probably RAM as well - during the 99% of time where it's not needed)
So wouldn't it make sense to make the expert models language specific here? Then you could take advantage of the fact that a language rarely changes inside a session and keep that expert in VRAM for the entire session. You could also avoid dragging parameters along with you for languages that are practically never used.
Like there is no explicit goal aside from each 'expert' getting roughly equal weight?
And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?
I do not know what I am talking about, this is my limited understanding...
Because the natural continuation of this is to say like, "Ok I want to load the bird detection expert and the navigation expert but leave the medieval European history expert behind", and my understanding is that this is not really how it works. At least at the moment.
was this meant to read "sharding"?
I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.
If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful.
If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.
If you had a deadly condition, and no diagnosis worked, and a specific model had the answer... past that I wouldn't use it.
the canary in the coal that this has not change is nvidia share price.
For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.
So what you get is 80x more expensive and you wait 416 days to get it.
I have all praise for those taking this on and in my idiom would call it *the lord's work."
The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.
It's work like this that moves things forward.
Hoping a winner emerges with some real momentum behind it.
now we'll just always be vulnerable from different angles in different and unique ways across the globe.
better? well, no.. but it is different, and biology has offered tons of wisdom about why it's a good thing to achieve things different ways across the world, maybe some far-future version of this weird fragmentation of work that is happening now will contribute to some kind of herd effect that reduces the severity or magnitude of some bug or work of malice.
who knows though, i'm probably just considering it too positively. The disintegration of consolidated effort is a pita, I agree.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity.
Also guessing that you need to be able to hold at least 3-layers at once in the memory, given M x N = R operation, M is the previous layer, N is next, and R is the result. on the next "layer", the R (result) becomes M, gets computed against the next layer, N, yielding the further result R'. And so on, until all layers are processed.
I assume it's horribly slow, but can be put in a non-intrusive background task...
It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.
To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which is not fast enough for comfortable interactive agentic use. A couple of new GPUs, like Radeon AI Pro 9700 at $1400 each, probably gets you fast enough for comfortable interactive use with small models like those. Those small models are not competitive with Claude models, however (maybe they beat Haiku sometimes). They can write a little Python or make a web page, they can't architect a real application.
To run a near-frontier model like Kimi K3 or GLM 5.2 at comfortable speeds, you need serious hardware with 768GB VRAM, minimum. I think Asus is releasing something like that for about $150k soon. You can run DeepSeek V4 Flash at almost comfortable speeds and in a decently capable quantization on two DGX Sparks or Asus GX10s (about $10,000).
Or, you could use DeepSeek V4 Flash from DeepSeek.com, at blistering speeds and with huge contexts, for something like a decade or two for that same $10,000.