Used RTX 3090 vs New Mid-Range: The Best Value GPU for Local AI

The second-hand market has settled on one recommendation for local inference, and it has not moved in a while: buy a used 3090. The reason is blunt — 24GB of VRAM for less money than any new card with that much. But a used RTX 3090 for local AI is not a free lunch, and the honest comparison against a new mid-range card is closer than the enthusiasts make out.

Why 24GB is the number

VRAM capacity is a hard wall. Below it everything runs at GPU speed; above it, layers spill to system RAM and generation slows to whatever your DDR can sustain. There is no graceful middle.

24GB is where a 32B-class model at 4-bit — call it 18GB of weights — fits with several gigabytes left for KV cache and overhead. A 16GB card does not, and a 12GB card is firmly in 14B territory. That one step is the difference between a model that is a toy and one that will take a real refactoring task seriously.

ollama run qwen2.5-coder:32b
ollama ps   # want to see 100% GPU, not a CPU/GPU split

What it costs to actually run

The card’s rated board power is 350W, and sustained generation will hold it near that. Work out your own number rather than accepting a forum figure:

# energy for an hour of sustained load, at your own tariff
python3 -c "w=350; hours=1; rate=0.25; print(f'{w/1000*hours*rate:.2f} per hour')"

Plug in your actual unit rate. For bursty interactive use it is negligible; for a box that batch-processes overnight, it adds up to a real line on the bill, and it is the strongest argument against leaving one idling as an always-on server. You also need a power supply with genuine headroom — transient spikes on this generation are notorious, and a marginal PSU shows up as a reboot mid-generation rather than a warning.

The good news is that you can give up very little to save a lot. Inference is bandwidth-bound, not compute-bound, so capping power costs far less throughput than you would expect:

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 280      # watts; measure before and after on your own workload
nvidia-smi --query-gpu=power.draw,temperature.gpu,memory.used --format=csv -l 2

I would not quote you a percentage, because it varies by model and backend. Measure it: run the same prompt at a couple of power limits and compare the tokens per second the runtime reports.

The risks you are actually taking

No warranty. That is the headline. These cards are several years old, many were mined on, and the ones that were mined on ran hot for a long time. The specific weak point on this design is the memory modules on the rear of the board — they sit against thermal pads that dry out, and degraded pads mean memory junction temperatures climbing under sustained load.

Checks to run in the first hour, while a return is still plausible:

• Confirm all the memory is visible and healthy — load a model that fills most of the 24GB and leave it generating for twenty minutes.
• Watch memory junction temperature, not just the core temperature, under that sustained load.
• Check for ECC or Xid errors in the kernel log after the soak.
• Physically inspect for bent fins, dust-packed heatsinks and any smell of scorched plastic.

nvidia-smi -q -d TEMPERATURE
nvidia-smi -q -d ECC
sudo dmesg | grep -i xid

Budget for a repaste and fresh thermal pads on any card that has clearly been worked. It is an afternoon and a modest parts cost, and it is the difference between a card that throttles and one that does not.

The case for buying new instead

A current mid-range card typically lands at 12GB or 16GB, draws considerably less power, is quieter, comes with a warranty and has newer tensor hardware with better support for low-precision formats. If your workload is 7B to 14B models — which covers autocomplete, chat, summarising and embeddings — it is the better buy, and it is the one I would recommend to anyone who does not want a hobby alongside their hardware.

What it cannot do is make 16GB into 24GB. No amount of newer architecture gets a 32B model into a card that cannot hold it. Capacity beats generation for this workload, every time.

Two cards?

Two 3090s give you 48GB and put 70B-class models at 4-bit within reach. It is a real option and llama.cpp splits layers across GPUs happily. It also doubles the power draw, needs a motherboard with the lanes and physical clearance, and turns a simple build into a project. Do it because you want 70B, not because it looks like better value per gigabyte.

What I’d buy

If you are running 32B-class models and you are comfortable opening a computer, the used 3090 is still the value pick and nothing has replaced it. Buy locally so you can test before money changes hands, assume you will repaste it, and cap the power limit on day one.

If you want a machine that just works, will sit in a quiet room, and will mostly run models up to 14B, buy the new mid-range card and spend the difference on RAM and a decent SSD. Prices on both sides of this comparison move constantly, so check the current second-hand asking price against a new card before committing — the gap is what makes the argument, and the gap changes.

Either way, work out the memory requirement first. If the plan is to drive the card from an agent, the context lengths in the complete guide to using Claude Code with local LLM models will push your KV cache well beyond what a casual chat session needs, and that is often what decides 16GB versus 24GB.