Skip to content

deploy.md

Latest commit

 

History

History
178 lines (122 loc) · 7.04 KB

deploy.md

File metadata and controls

178 lines (122 loc) · 7.04 KB
Raw

Deploy flashbulb on a free host

This guide puts a live instance on Oracle Cloud Always Free at no cost. It replaces the retired Alibaba Cloud ECS deployment.

Audience: you can use SSH and a cloud console. You do not need Docker or Kubernetes.

Result: a public URL that serves the dashboard, with a live model answerer, for $0 per month.

What you need first

  • An Oracle Cloud account. The Always Free tier needs a card for identity checks. It does not charge you.
  • A free NVIDIA API key, for the answerer. Step 1 covers this.
  • About 40 minutes. Most of that is the PyTorch install.

Why this host

The app loads PyTorch, a sentence embedder, and an NLI cross-encoder. It needs about 1 GB of RAM at startup. That rules out every 512 MB free tier, such as Render and Koyeb. Hugging Face Spaces no longer offers free Docker or Gradio Spaces. Oracle Always Free stays on permanently and gives enough RAM.

Oracle keeps changing what Always Free includes. Take whatever the console currently offers. 1 OCPU and 6 GB is enough. The app needs about 1 GB, so 6 GB leaves ample headroom. One core is fine for a low-traffic demo, though startup takes longer while the models load.

Step 1: Get a free model endpoint

The app is provider-agnostic. It speaks the OpenAI-compatible protocol through llm_client.py. Any provider works with no code change. This guide uses NVIDIA's hosted catalog, which is free and needs no credit card.

Warning: the read path uses native function-calling. Choose a model that supports tool use. A model without tool support degrades the recall step.

  1. Create a free account at https://build.nvidia.com. No credit card is needed.
  2. Generate an API key. It starts with nvapi-.
  3. Use meta/llama-3.3-70b-instruct, or pick another tool-capable model from the catalog.

To list exactly what your key can reach:

curl -s -H "Authorization: Bearer $LLM_API_KEY" \
  https://integrate.api.nvidia.com/v1/models | grep -o '"id":"[^"]*"'

Model IDs rotate. Read the ID from that listing. Do not copy it from a blog post.

Note: the hackathon build answered with Qwen on Alibaba DashScope. No free Qwen text endpoint exists any more, so this guide uses a different model. The memory controller is unaffected: it calls no model at all.

Step 2: Create the Oracle instance

Warning: Oracle keeps cutting this tier without announcing it. It went from 4 OCPU and 24 GB to 2 OCPU and 12 GB on 2026-06-15, and some tenancies now offer only 1 OCPU and 6 GB. Take what the console gives you. Do not size anything from a number you read elsewhere.

Warning: the Ampere A1 shape is often out of capacity in US regions. Do not pick a US region if another one suits you. Frankfurt and Singapore usually provision quickly.

  1. Create a Compute instance.
  2. Select the shape VM.Standard.A1.Flex. This shape is ARM (aarch64).
  3. Set the memory to the Always Free maximum your console offers. 6 GB is enough.
  4. Select an Ubuntu 22.04 image or later.
  5. Save the SSH private key that the console offers. You cannot download it again.
  6. Create the instance. Record the public IP address.

If the console reports "Out of host capacity", retry later or select another region.

Step 3: Open port 80 in the VCN

Warning: this step alone does not make the port reachable. The instance also has a local firewall. Step 4 opens that one. Skipping either step leaves the site unreachable.

  1. Open the instance's subnet. Open its security list.
  2. Add an ingress rule.
  3. Set source CIDR 0.0.0.0/0, protocol TCP, destination port 80.
  4. Save the rule.

Step 4: Run the setup script

The script installs the dependencies, warms the model cache, opens the local firewall, and starts the service. It is idempotent, so you can run it again safely.

ssh -i <your-key.pem> ubuntu@<PUBLIC_IP>
curl -fsSL https://raw.githubusercontent.com/somtri/flashbulb/main/deploy/setup.sh | bash

The PyTorch install takes several minutes. The script prints a numbered progress line per step.

Step 5: Add your key

The script creates /opt/flashbulb/.env from the template. Edit it now.

nano /opt/flashbulb/.env

Set these three values:

LLM_BASE_URL=https://integrate.api.nvidia.com/v1
LLM_API_KEY=nvapi-your-key-here
LLM_MODEL=meta/llama-3.3-70b-instruct

Restart the service to load them:

sudo systemctl restart flashbulb

If you leave the key blank, the dashboard still runs. It falls back to MockLLM and reports its token counts as synthetic.

Step 6: Verify

Run this from your own machine, not from the server:

curl -sS -o /dev/null -w '%{http_code}\n' http://<PUBLIC_IP>/

A 200 means the deployment works. Then open the URL in a browser and confirm three things:

  1. A chat turn returns a reply.
  2. The write-token counter stays at 0.
  3. Snapshot, Reset, and Restore round-trip the memory.

If curl hangs instead of returning, the firewall is the cause. Recheck steps 3 and 4.

Operating it

sudo journalctl -u flashbulb -f        # follow the logs
sudo systemctl restart flashbulb       # restart
systemctl show flashbulb -p MemoryCurrent   # current memory use

The memory cap matters

This app leaks about 300 MB per day. The retired ECS box wedged twice because of it, on 2026-07-29 and 2026-08-15. On a host with no swap the kernel evicts all page cache to feed the leak. Every process then jams waiting on disk, while the kernel keeps answering TCP. The machine looks alive from outside and is dead inside, and only a forced power cycle recovers it.

deploy/flashbulb.service sets MemoryMax=3G and Restart=always. The kernel therefore kills only this service and systemd restarts it in seconds. The host stays healthy. The setup script also adds a 2 GB swapfile, because the absence of swap is part of how the old box died.

Warning: never set MemoryMax below 2G. The app needs about 1 GB just to load its models. A cap near that floor kills it during startup, and systemd then restarts it into the same wall forever. On a 6 GB box, 3G is correct.

Warning: do not remove the memory limits. Without them a leak of this kind takes down the whole machine, and a graceful console restart cannot recover it.

A restart drops the in-memory demo sessions. It does not affect snapshots, which are files on disk.

Add an uptime check

Nothing on the box can report that the box is down. Create a free external check at https://uptimerobot.com. Point it at your URL every 5 minutes. Send alerts to SMS or push, not email.

Retire the Alibaba deployment

Do this only after the new URL returns 200.

  1. Delete the ECS instance i-t4n6mvzior577y26qylg in ap-southeast-1.
  2. Delete the OSS bucket, after you confirm you want to lose the stored snapshots.
  3. Delete the RAM user whose AccessKey the app used.
  4. Delete the DashScope API key.
  5. Check the billing page for any remaining active resource.
  6. Delete the local fl.pem SSH key, which now opens nothing.

Warning: deleting the OSS bucket destroys every stored memory snapshot permanently. Download anything you want to keep first.