What's a good offline AI cooder thing, and what kind of resources does it need to run?
There are various LLMs trained for coding but for the sake of an example, Claude Sonnet. You would run this locally and then you can query the model in various ways such as direct from the terminal, in the browser, via a plug-in to your editor (you might use the Continue extension in VS Code for example, or if using Claude specificially, their own VS Code extension). The simplest way to get and down models as a start would be LM Studio which gives you a self-contained interface for downloading models from public repositories, running them and querying them with your own prompts. You might attach a program file to a query. Or you can set up local file access to chosen directories and make it aware of them so it can go looking, cross-reference files, etc. For serious local coding you would probably want a plug-in to your editor, like the VS Code extensions mentioned. LM Studio can run the model as a local service that other tools can connect to.
In terms of hardware, you need both system RAM and VRAM with the latter usually being the constraint. 8GB VRAM would be your lowest viable minimum though that's low for this. In addition to loading the model, you want space for the Model Context. Kind of how many tokens it can handle. More tokens = More Lines of Code (both in and out).
Most models are available in different sizes, formally classed by the number of parameters. So you might get an 7 billion parameter model (7B), a 13 billion parameter model (13B) or some wopping 70B model (about as high as you can feasibly run on consumer hardware, and that is pushing it). The higher the number the more the model "knows". And the correspondingly more about of VRAM required to load it. (You can technically off-load models to system ram, splitting them up. Really not ideal). The same model is often available in different versions.
There is another axis which is quantisation. That's a way of reducing the size of a model but at the cost of making it more... fuzzy. So its accuracy gets a little less. A small amount of quantisation is supposed to be fine. More extreme quantisation, gets worse and worse. If the parameters is the knowledge of the model, the quantization is the degree of Alzheimers it has.
For running a coding model effectively locally I'd say 16GB VRAM is your minimum. 20GB VRAM and above is better. You can do this on AMD hardware (I wouldn't try lower than RDNA3 and RDNA4 is significantly better for LLMs) but it's indisputable that NVIDIA hardware is currently better.
You can also do semi-private coding assistance by running the model in the cloud somewhere (Azure, AWS and Google all have out of the box ways to configure and run this) or you could rent some compute on something like Runpod and roll your own. So long as your local tools (e.g. VS Code) can connect to it and authenticate, you're alright. Maybe some lag but I don't think generally it would be an issue - you're transferring code files not big blobs of binary. This is obviously not "local", but it's also not providing Anthropic or OpenAI all your code for them to parse and use as training data. So it's a cost effective way to dip your toe in the water if you don't have a grand or more of GPU lying around.
I'm currently weighing up the pros and cons of getting a new GPU for this. The top of the line consumer level would be the 5090 which has 32GB VRAM (large enough for a good coding model and the Context it needs to parse and generate code) and the Blackwell architecture which is current leader. Cost is horrifying though and it has a thirst for power that would shame Hilary Clinton. The lower options that have 16GB VRAM probably aren't quite enough for me. So there is the AMD Radeon Pro 9700 which has 32GB VRAM and is less than half the cost of the 5090. But also less than half the speed at these tasks. It's probably enough for my purposes, but I'm trying to find good benchmarks.
I hope all this is helpful. I've seen enough of your posts to know you have a lot of technical knowledge so I tried to just give you the high level context approach you could drill down from.
But I'm new at this.