A record of setting up SD1.5 + ControlNet on a GTX 1660 SUPER (6GB VRAM). What tripped me up wasn't a VRAM shortage — it was three things: "RAM spiking to ~26GB and OOMing," "all-black NaN images under fp16," and "a 3.2x slowdown from forcing the math SDPA backend." Later I also discovered that every one of my step timings had been distorted by spilling. Causes and fixes, all of it.
What you'll get from this post
- An SD1.5 configuration where ControlNet runs on a 6GB GPU
- The fp16 NaN problem on GTX 16xx (TU116) cards and how to work around it
- Why sequential_cpu_offload eats RAM rather than VRAM
- Final speeds: measured figures of ~5 s/image at Hyper-SD15 1-step and ~1 min/image at a full 25 steps
What I used
- MSI GeForce GTX 1660 SUPER GAMING X 6GB (check on Amazon) — the GPU used here
- CFD DDR4-3200 32GB desktop memory (check on Amazon) — I hit a RAM OOM, so I'd recommend 32GB or more
- WSL2 (Ubuntu 22.04), Python 3.12, diffusers 0.38
- Realistic Vision V6.0 B1 (noVAE) checkpoint
- ControlNet (OpenPose / Canny / Depth) + LCM-LoRA / Hyper-SD15
Background: I killed SDXL ControlNet after 1 hour and 13 minutes
Originally I was implementing ControlNet in a different environment, built on SDXL. The implementation finished running at 21:40 on 2026-04-30 — and by 22:53 I had committed its removal. One hour and thirteen minutes.
The cause was RAM. With enable_sequential_cpu_offload under SDXL, peak VRAM stays around 4GB. But with that mechanism the entire model stays resident in RAM. Add ControlNet on top (~5GB in fp32) and:
| Configuration | Peak VRAM | Resident RAM | Result |
|---|---|---|---|
| SDXL fp32 + FaceID | ~4GB | ~21GB | ✅ (on a 24GB machine) |
| + add ControlNet | ~4GB (unchanged) | ~26GB ❌ | OOM |
VRAM was never the problem. The cost of adding ControlNet lands on the RAM side. "It fits in 6GB of VRAM, so why the OOM?" only makes sense once you understand how sequential_cpu_offload works.
How sequential_cpu_offload actually works
The diffusers enable_sequential_cpu_offload moves the UNet's ~700 submodules into VRAM one at a time, runs inference, and moves each back to RAM — repeating that every step. The whole model stays resident in RAM. That's 700 CPU↔GPU round trips per step, with the Python GIL pinning one core at 100%.
SD1.5 is ~4GB total in fp32. It fits in 6GB of VRAM, so sequential_cpu_offload becomes unnecessary. That was the motivation for standing up a separate SD1.5 environment.
The three things that tripped me up
1. All-black NaN images under fp16 (a known GTX 16xx problem)
I ran a microbenchmark across five configurations:
| Configuration | Time per step | Output |
|---|---|---|
| T1: SD1.5 fp16, no CN, math SDPA | 15.8 s/step | all black, 1224 bytes (NaN) |
| T3: + 1 ControlNet, math SDPA | 22.6 s/step | NaN |
| T4: + flash + mem-efficient SDPA | 7.1 s/step | NaN (faster, but still) |
| switched to fp32 (default SDPA) | see below | ✅ generates correctly |
Changing the SDPA backend didn't make the NaNs go away. The cause is that the GTX 1660 SUPER (TU116) has no Tensor Cores. The fp16 softmax in the attention layer underflows and overflows. It's the same problem that AUTOMATIC1111 ships an official --no-half flag for on GTX 16xx cards. Switching to fp32 fixed it immediately.
2. Forcing the math SDPA backend was causing the 3.2x slowdown
The gap between T3 (22.6 s/step) and T4 (7.1 s/step). The culprit was code carried over from the other environment:
# inherited from the other environment (written as an fp16 NaN workaround)
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_mem_efficient_sdp(False)
torch.backends.cuda.enable_math_sdp(True)Deleting this and letting PyTorch pick the backend automatically gives a 3.2x improvement. That said, flash SDPA is fp16/bf16 only, so once you move to fp32 you're back on the math kernel anyway. "Delete the forcing code and move to fp32" is the correct combination.
3. Every measurement had been distorted by spilling (discovered later)
After the environment was up, I started emitting a vram_peak_mb value into the generated images' metadata — and found that fp32 with one ControlNet was peaking at 6,726 MB. The effective ceiling for a GTX 1660 SUPER under WSL2 is about 6,100 MB.
Everything had been spilling. The early numbers (13.7 s/step, 5 min 55 s for a full 25 steps) were all measured while WSL2 was spilling into host RAM.
The fix is enable_model_cpu_offload (per-model, 5–10 CPU↔GPU swaps):
| Configuration | --offload none (spilling) | --offload model (current default) | Speedup |
|---|---|---|---|
| Full 25-step + 1 CN | 584.7s / VRAM 6,726 MB | 61.5s / VRAM 5,298 MB | 9.5x |
| Hyper-SD15 4-step + 1 CN | 62.9s / VRAM 6,680 MB | 11.1s / VRAM 5,146 MB | 5.7x |
| Hyper-SD15 1-step | 32.4s / VRAM 6,680 MB | 4.9s / VRAM 5,106 MB | 6.6x |
| Multi-CN (2) 25-step | 884.6s / VRAM 8,124 MB | 680.1s / VRAM 6,694 MB | 1.3x (still spilling) |
With a single ControlNet, VRAM settles at 5.1–5.3GB and stops spilling, which is where the 9.5x comes from. With two ControlNets, --offload model still leaves you at 6.7GB and spilling, so the gain is limited.
Low-step distillation for SD1.5: what plays the role of SDXL Lightning?
Lightning, which I'd been using on SDXL, has no SD1.5 equivalent — ByteDance only released it for SDXL. The SD1.5 options:
- LCM-LoRA (
--lcm): 4 steps, ~10 s/image - Hyper-SD15 CFG-distilled (
--hyper {1,2,4,8}): ~5 s at 1 step. Negative prompts have no effect - Hyper-SD15 CFG-preserved (
--hyper-cfg {8,12}): ~4.5 min at 8 steps. Negative prompts work, and it's less prone to oversaturation
For exploring poses I run large batches at --hyper 1 (5 s/image); for finishing portrait shots, --hyper-cfg 8 (with negative prompts working) is the easier tool.
How the two environments divide the work (final form)
| SD1.5 environment | SDXL environment | |
|---|---|---|
| Primary use | exploring poses and composition | final output, FaceID portraits |
| Speed (full 25 steps) | ~1 min/image (61.5s measured) | ~16 min/image |
| Speed (low-step distilled) | Hyper 1-step ~5 s | Lightning 8-step ~6–7 min |
| Resolution | 512×768 (native) | 1280×720 / 832×1216 |
| ControlNet | OpenPose / Canny / Depth ✅ | ❌ (removed due to RAM limits) |
The workflow I'd recommend: use --hyper 1 on SD1.5 (5 s/image) to burn through a lot of pose and composition attempts, then hand the chosen reference image over to the SDXL environment for the final output. SD1.5's OpenPose skeleton images can't be used directly with SDXL's ControlNet models — the latent spaces differ — but you can bridge the gap by passing the original reference image in as IP-Adapter input.
FAQ
Q. Is there any way to use fp16 on a GTX 1660 SUPER?
A. For now fp32 is the only option. TU116 has no Tensor Cores, and the fp16 softmax in the attention layer underflows and overflows. It's the same known issue behind AUTOMATIC1111's --no-half and ComfyUI's --force-fp32. On RTX 3060 or later (Ampere), fp16 works and flash SDPA should get you to 1–2 s/step.
Q. Is there a way to add ControlNet to SDXL after all?
A. A configuration without FaceID (txt2img + ControlNet only) runs in about 19GB of RAM. Using FaceID and ControlNet together pushes past ~26GB, so splitting that work off to SD1.5 is the realistic choice.
Q. How long does Multi-ControlNet (two at once) take?
A. About 11 min/image, because even with --offload model you're at 6.7GB of VRAM and still spilling. The gain over a single ControlNet is limited precisely because the spilling isn't fully eliminated.
※ The steps and figures in this post were verified in a May 2026 environment (diffusers 0.38, GTX 1660 SUPER, WSL2 + Ubuntu 22.04). Behavior may change with different library or GPU driver versions. If something doesn't work, let me know in the comments.
Wrap-up
I built an SD1.5 + ControlNet environment on a GTX 1660 SUPER (6GB VRAM, 24GB RAM). After three pitfalls — fp16 NaNs, the slowdown from forcing math SDPA, and measurements distorted by VRAM spilling — I ended up with a setup that does ~5 s/image at Hyper-SD15 1-step and ~1 min/image at a full 25 steps. Splitting duties with the SDXL environment (16 min/image) makes pose exploration much faster.
If this post was useful, I'd be glad if you shared it on X.
App by the author of this blog
I made an iOS reading management app called My Bookstore. Simple bookshelf management — give it a try.
No comments:
Post a Comment