Cosmosage Unshelver: GPU capacity shortage on Jetstream2, UI stuck at 0%

[zonca]

Problem

The Cosmosage chat interface at https://cosmosage.phy240259.projects.jetstream-cloud.org/ is currently non-functional — the llama.cpp server web UI loads but generation stays stuck at 0% progress and never produces output.

Root Cause: GPU capacity shortage on Jetstream2

The original cosmosage_70b_zonca instance (flavor g3.xl = 1× full A100 40GB, 32 vCPU, 120GB RAM) could not be unshelved because Jetstream2 has no available g3.xl GPU hosts in the IU region. Every unshelve attempt since May 18, 2026 failed at the OpenStack scheduler with:

No valid host was found. There are not enough hosts available.

The scheduler error occurs at schedule_instances — this is not a transient failure. The 90 A100 GPU nodes (360 A100 SXM4 40GB GPUs) are fully occupied.

What was done

1. Old instance deleted — The stuck g3.xl instance could not be unshelved, resized, or migrated (all blocked by OpenStack policy for SHELVED_OFFLOADED instances). After attempting to recreate it on g3.xl (which also failed with the same scheduling error), the original boot volume was unfortunately lost (the volume attachment had delete_on_termination=True).

2. New instance created on g3.medium — This smaller GPU flavor (3× A100X-10C = 30GB total VRAM, 8 vCPU, 30GB RAM) does have available capacity on Jetstream2. The instance is now ACTIVE with floating IP 149.165.155.205.

3. Model weights recovered — The llmstorage volume (100GB) from the old llm_unshelve instance was detached and attached to the new instance. It contains Meta-Llama-3.1-70B-Instruct-Q3_K_L.gguf (35GB, Q3_K_L quantization, 70.5B parameters).

4. llama.cpp server running — Built llama.cpp with CUDA support, running on port 8080. The /health endpoint returns {"status":"ok"} and the web chat UI loads at the root path.

Current issue: generation stuck at 0%

The llama-server is configured with -ngl 40 (40 of ~81 layers on GPU, rest on CPU) because the 35GB model cannot fully fit in 30GB VRAM. With only 8 vCPU and 30GB RAM on g3.medium (vs 32 vCPU / 120GB RAM on the old g3.xl), CPU-offloaded inference is extremely slow — the UI shows 0% and appears frozen because token generation is happening but at an impractically slow rate.

The Q3_K_L 70B model is too large for the g3.medium flavor to run effectively with partial GPU offloading.

Possible solutions

1. Use a smaller quantization — A Q2_K or IQ2_XS quantization of Llama-3.1-70B would be ~20-25GB and could fit entirely in the 30GB VRAM, avoiding CPU offloading entirely. Quality would degrade but inference would be fast.

2. Use a smaller model — cosmosage-v3.1 (based on a smaller base model) could run entirely in VRAM and would be faster.

3. Wait for g3.xl capacity — When A100 hosts free up on Jetstream2, recreate the instance with the full A100 flavor. There is no ETA for this.

4. Use g3.large (16 vCPU, 60GB RAM, A100X-20C = 2×10GB slices = 20GB VRAM) — More CPU/RAM for offloading but still insufficient VRAM for the full model.

5. Obtain the original setup scripts — The boot volume with the full server configuration (inference server, systemd services, model download scripts) was lost. If the original setup is documented or scripted, we can rebuild properly with whichever flavor/model combination is chosen.

Infrastructure details

	Old (lost)	Current
Flavor	g3.xl	g3.medium
GPU	1× A100 40GB	3× A100X-10C (30GB total)
vCPU	32	8
RAM	120GB	30GB
Model	Q3_K_L 70B (35GB)	Same (recovered from llmstorage)
Inference	Unknown (vLLM?)	llama.cpp with partial GPU offload
Instance ID	814d3dd3-... (deleted)	a9220691-3479-4358-a2c8-c7298b16ac30

Action items

• Decide on model/quantization that fits g3.medium VRAM
• Recreate the inference server setup (original config lost with boot volume)
• Set up systemd service for llama-server so it survives reboots
• Test shelve/unshelve cycle with g3.medium (should work since this flavor has capacity)
• Update the unshelver controller config if needed

cc @tijmen

[tijmen]

Great find! I'd love to know where you found the error message ("No valid host was...")

Can we fit this model? (27.6 GB) https://huggingface.co/mradermacher/AstroSage-70B-20251009-i1-GGUF/resolve/main/AstroSage-70B-20251009.i1-IQ3_XXS.gguf

[zonca]

Update: backend is completely non-functional

I tested the llama.cpp server directly via SSH and the situation is worse than I initially described — inference is not slow, it is completely broken.

Direct API test

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Meta-Llama-3.1-70B-Instruct-Q3_K_L.gguf","messages":[{"role":"user","content":"Hello, what is 2+2?"}],"max_tokens":50}' \
  --max-time 300

Timed out after 5 minutes with zero response. No tokens generated at all.

Server logs

When a request hits the server, the logs show:

slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
stopping wait for next result due to should_stop condition (adjust the --timeout argument if needed)
cancel task, id_task = 0
slot release: id  0 | task 0 | stop processing: n_tokens = 92, truncated = 0

The server processes ~92 prompt tokens then hits an internal timeout and cancels. No completion tokens are ever produced.

Resource usage

Process: 18.6 GB RAM (60.6% of 29GB), using swap
GPU: 3× GRID A100X-10C (10GB each = 30GB total VRAM)
Config: -ngl 40 (40 of ~81 layers on GPU, 41 on CPU)
Context: -c 2048, --parallel 1, -t 8

The 35GB Q3_K_L model does not fit in 30GB VRAM. With 41 layers offloaded to CPU and only 8 vCPU / 29GB RAM (plus swap), the CPU-bound layers are so slow that the server times out before producing a single token. This is not a "slow but works" situation — it cannot generate any output at all on g3.medium.

Conclusion

The current setup is non-viable. The Q3_K_L 70B model requires a full A100 (40GB VRAM) or a smaller quantization that fits entirely in the 30GB VRAM of g3.medium. Partial GPU offloading with this hardware configuration does not work.

[zonca]

@tijmen — pinging you on this. The cosmosage chat is completely down. The original g3.xl instance could not be unshelved (no A100 hosts available on Jetstream2 since May 18), and in the process of trying to recover, the old boot volume was lost.

I rebuilt on g3.medium (3× A100X-10C = 30GB VRAM) and recovered the model weights from the llmstorage volume, but the Q3_K_L 70B GGUF (35GB) cannot run on this hardware — inference times out with zero tokens produced because 41 of 81 layers are CPU-offloaded on only 8 vCPU / 29GB RAM.

To get cosmosage back up we need either:

• A smaller quantization (Q2_K or IQ2) that fits entirely in 30GB VRAM
• Or wait for g3.xl capacity to free up on Jetstream2

Do you have the original setup scripts or know what quantization/inference server was running before? Also, do you have a preference between running a lower-quality quant on g3.medium now vs waiting for full A100 capacity?

[zonca]

Where the "No valid host" error came from

I queried the OpenStack instance actions API (microversion 2.51+) directly on the controller. The relevant call:

import requests
headers = {
    "X-Auth-Token": token,
    "OpenStack-API-Version": "compute 2.51",
}
resp = requests.get(
    f"https://js2.jetstream-cloud.org:8774/v2.1/servers/{SERVER_ID}/os-instance-actions",
    headers=headers,
)
actions = resp.json()["instanceActions"]
# Then for each action:
resp = requests.get(
    f".../os-instance-actions/{ACTION_ID}",
    headers=headers,
)
# The events array shows: "event": "schedule_instances", "result": "Error"

Every unshelve attempt since May 18 had schedule_instances → Error. The nova-scheduler.log on the Jetstream2 side would have the full "No valid host was found" message, but that is admin-only. The instance fault field (server.fault) was None — the error only surfaces through the actions API.

AstroSage-70B IQ3_XXS (27.6 GB)

Yes, that should fit entirely in 30GB VRAM! At 27.6 GB with 30 GB available, there is enough headroom for the KV cache too. This would let us run with -ngl 99 (all layers on GPU) and avoid the CPU offloading problem entirely.

Want me to download it and swap it in? I can:

1. Kill the current llama-server
2. Download AstroSage-70B-20251009.i1-IQ3_XXS.gguf to the llmstorage volume
3. Restart the server with full GPU offloading (-ngl 99)

[tijmen]

Yes, please. Much appreciated!

[zonca]

Update: cosmosage-v3 is running but GPU is broken on g3.medium

I got cosmosage-v3 Q4_K_M (7B, 4.6GB) running on the instance. It works on CPU only at ~1.7 tokens/second and generates correct cosmology answers:

Q: What is dark energy?
A: Dark energy is a mysterious component of the universe that is causing its expansion to accelerate. It is thought to make up about 68% of the total energy of the universe.

GPU inference is broken on g3.medium

The g3.medium flavor gives 2× GRID A100X-10C vGPUs (10GB each). CUDA works — PyTorch can allocate and compute on both GPUs. However, llama.cpp GPU inference is unusable:

Config	Speed	Status
CPU-only (-ngl 0)	1.7 tok/s	Works
Single GPU (-ngl 17, CUDA_VISIBLE_DEVICES=0)	~0.03 tok/s	50x slower than CPU!
Both GPUs (-ngl 33)	~0.03 tok/s	Same — unusable

The GRID A100X-10C vGPU profile only allocates 10 compute cores out of the 108 on a physical A100. The virtualization overhead makes CUDA compute slower than CPU — a known issue with fractional vGPU profiles on Jetstream2.

What about AstroSage-70B?

The 26GB AstroSage model is downloaded on the llmstorage volume, but it cannot run on g3.medium:

• GPU: too slow (same vGPU issue)
• CPU: 70B model needs 40+GB RAM; g3.medium has only 29GB

Bottom line

cosmosage-v3 (7B) is live at https://cosmosage.phy240259.projects.jetstream-cloud.org/ on CPU at ~1.7 tok/s. A full A100 (g3.xl) is needed for either:

• 70B models at usable speed
• Or any model with working GPU acceleration

The chat UI works but responses take ~30 seconds for a short answer.

[zonca]

Status Update

I have an automated script retrying every 2 minutes to grab a g3.xl instance (full A100 40GB) on Jetstream2 IU, but so far no success — the GPU capacity is fully saturated.

Current state:

• cosmosage-v3 (7B) on CPU is live as a temporary fallback at https://cosmosage.phy240259.projects.jetstream-cloud.org/ (~1.7 tok/s)
• AstroSage-70B (your suggested model) is already downloaded on the llmstorage volume, ready to deploy once we get a full A100
• The g3.medium flavor (fractional vGPU) has broken CUDA performance — GPU inference is 50x slower than CPU due to vGPU virtualization overhead

Waiting on: a g3.xl instance to become available so we can run the 70B model with proper GPU acceleration.

[tijmen]

That sounds good. Fingers crossed a g3.xl instance frees up. I think a reasonable solution here is to grab a g3.xl instance as soon as we can and keep it active without shelving/unshelving. This will burn at 84 SU/hr which will last us 2 months, after which we can close this project.

The model should be
https://huggingface.co/AstroMLab/AstroSage-70B-20251009
or one of its quants from
https://huggingface.co/mradermacher/AstroSage-70B-20251009-i1-GGUF
https://huggingface.co/mradermacher/AstroSage-70B-20251009-GGUF

[zonca]

Update: AstroSage-70B is LIVE on a full A100! 🎉

After ~37 retry attempts over ~1.5 hours, we grabbed a g3.xl instance with a full A100-SXM4-40GB.

AstroSage-70B-20251009.i1-IQ3_XXS is now running at:

Metric	Speed
Prompt processing	224 tok/s
Generation	25 tok/s
Model size	26GB (fits entirely in 40GB VRAM)

The chat UI is live at https://cosmosage.phy240259.projects.jetstream-cloud.org/

All layers are on GPU (-ngl 99), no CPU offloading. Responses are snappy — a 100-token answer completes in ~4 seconds.

[zonca]

Final update: AstroSage-70B running on full A100, keeping this model

We got a g3.xl instance with a full A100-SXM4-40GB. AstroSage-70B IQ3_XXS is running at ~25 tok/s — working great.

Why we kept AstroSage-70B instead of the larger Q3_K_L model:

• The Q3_K_L model is 35GB — too large for the 40GB A100 (no room for KV cache)
• AstroSage-70B IQ3_XXS (26GB) fits with ~12GB headroom for KV cache and context
• The IQ3_XXS quantization is the one designed for constrained VRAM, and it works well

Still TODO on the server side:

• Set up systemd service for llama-server (survive reboots/unshelves)
• Add llmstorage volume mount to /etc/fstab
• Update the unshelver app config for the new instance ID

The service is live at https://cosmosage.phy240259.projects.jetstream-cloud.org/

[zonca]

Latest status

AstroSage-70B-20251009.i1-IQ3_XXS is running on a g3.xl (full A100-SXM4-40GB) at ~25 tok/s.

Live at: https://cosmosage.phy240259.projects.jetstream-cloud.org/

Key details:

• Instance ID: 2f755193-31fa-46b4-be60-747e0e75b21a (g3.xl, IU region)
• Floating IP: 149.165.155.205
• Model on Cinder volume llmstorage (attached as /dev/sdb)
• The Q3_K_L model (35GB) is too large for the 40GB A100 — no room for KV cache, so we went with AstroSage-70B IQ3_XXS (26GB)
• Fractional vGPU flavors (g3.medium, g3.large) have broken CUDA performance — 50x slower than CPU due to vGPU virtualization overhead. Only g3.xl (full A100 passthrough) works
• GPU capacity on Jetstream2 IU is very tight — g3.xl instances are rarely available, and unshelving often fails with "No valid host"
• I prefer to keep the unshelver app running so the instance auto-recovers when capacity frees up

Still TODO on the server:

• Set up systemd service for llama-server (auto-start on unshelve/boot)
• Add llmstorage mount to /etc/fstab
• Configure unshelver to send email notification if unshelving fails (so I know to check manually)

[zonca]

Email notifications configured

The unshelver app now sends email to zonca@sdsc.edu whenever an unshelve attempt fails (e.g., GPU capacity shortage). This uses Postfix on the controller VM to relay directly to inbound.ucsd.edu.

Tested and confirmed working — notification emails are delivered successfully.

[zonca]

Full shelve/unshelve cycle now automated ✅

The entire wake-up workflow is now hands-free:

1. User clicks "Wake Cosmosage" on the launcher page
2. Unshelver brings the instance back to ACTIVE
3. fstab entry auto-mounts the llmstorage Cinder volume (/dev/sdb → /mnt/llmstorage)
4. systemd service (llama-server.service) auto-starts llama-server with AstroSage-70B
5. Health check confirms the service is responding
6. Chat UI is live at https://cosmosage.phy240259.projects.jetstream-cloud.org/

Tested end-to-end: shelved instance → unshelve via webapp → volume auto-mounted → llama-server auto-started → correct cosmology answers at ~25 tok/s.

Email notifications are also working — zonca@sdsc.edu gets notified if unshelve fails (typically GPU capacity shortage on Jetstream2 IU).

[tijmen]

Thanks Andrea. Looks good in my testing.