There will be times when the struggle seems impossible. Alone, unsure, dwarfed by the scale of the enemy. Freedom is a pure idea. It occurs spontaneously and without instruction. Random acts of insurrection are occurring constantly. There are whole armies, battalions that have no idea that they’ve already enlisted in the cause. And even the smallest act of insurrection pushes our lines forward. Tyranny requires constant effort. It breaks, it leaks. Authority is brittle. Oppression is the mask of fear.
16GB is plenty for even older model setups. Now they’ve got a few models designed so you load just parts of the model onto the GPU (Mixture of Experts) and use the CPU for less referenced sections, so you get both reasonable speed and a much more complex model.
Narrator: The fire detector in fact did not work.
There’s more than one kind of backup in life, check them all.
Most models are going to require CUDA. There are some AMD ones out there, but it’s a totally different math and setup. As for the one I mentioned, it’s a pretty new idea so there are only a few out there, maybe just one (Qwen based). But I did get a 31B model to work on my 12GB, I just had to move from Ollama to llama.cpp to gain the control needed to set the parameters, and fine tune what it put on the CUDA to the max it would take. I had Claude help me along the way.
It’s new enough that there aren’t any good abliterated/uncensored models yet.