Adopt existing LXC containers into pve-compose management

[PoBruno]

Right now pve-compose only manages containers it created itself. If you already have LXC containers running Docker Compose stacks, there is no way to bring them under pve-compose control.

On my host I have 17 LXC containers. 10 of them run Docker Compose stacks with bind mounts pointing to app directories on the host. Only 1 has a lxc.json file (the one pve-compose created). The other 9 were set up manually or with scripts, so pve-compose ps, pve-compose doctor, and every other command just fails with "no lxc.json found".

The real data looks like this.

CTID  HOSTNAME       MP0 SOURCE                        MP0 TARGET
117   navidrome      /data/app/navidrome/               /mnt/navidrome
200   immich         /data/app/immich                   /mnt/immich
201   owncloud       /data/app/nextcloud                /mnt/data
203   n8n            /data/app/n8n                      /mnt/n8n
210   gitea          /data/app/gitea                    /mnt/gitea
212   jellyfin       /mnt/pve/storage/app/jellyfin      /mnt/data
241   casa360-prod   /data/app/casa360/prod             /mnt/prod
242   casa360-dev    /data/app/casa360/dev              /mnt/dev

Only speedtest has a lxc.json. The 9 others are invisible to pve-compose.

The idea is to add a command that reads the real container config from /etc/pve/lxc/<ctid>.conf and generates a valid lxc.json from it. After that, all pve-compose commands work normally.

Several edge cases already visible in the data above. CT 201 has hostname owncloud but the directory is called nextcloud. CT 212 uses a completely different storage path. CT 117 has a trailing slash in the mount source. CT 241 and 242 live in nested subdirectories.

[PoBruno]

The core question is how to find which container belongs to the current directory.

First thought was matching by hostname. pve-compose uses basename $PWD as hostname, so cd /data/app/immich means hostname immich. Just grep the conf files for hostname: immich and you get the CTID.

But that breaks immediately. CT 201 has hostname owncloud and the directory is /data/app/nextcloud. hostname matching fails here.

Better approach: match by the bind mount source path. Every container that runs a compose stack has an mp0 line in its conf file pointing to the host directory. That is the real link between directory and container. Not the hostname.

mp0: /data/app/immich,mp=/mnt/immich

The source part before ,mp= is the host path. If that matches $PWD, we found the container.

Algorithm:

1. normalize $PWD (strip trailing slash)
2. for each /etc/pve/lxc/*.conf
   a. extract mp0 source (text before ,mp=)
   b. normalize (strip trailing slash)
   c. if source == $PWD, thats our match
3. one match: return CTID
4. zero matches: fallback to hostname grep
5. multiple matches: error out

This handles all the edge cases. Trailing slash on CT 117, nested paths on CT 241/242, different storage on CT 212. The path is the path.

[PoBruno]

Once we know the CTID, we need to extract every field from the conf file and build a valid lxc.json.

The conf format is simple. One key per line, colon separated. Some fields are compound (comma separated key=value pairs inside one line).

Simple fields are trivial:

hostname: immich          -> "hostname": "immich"
cores: 3                  -> "cores": 3
memory: 4096              -> "memory": 4096
swap: 0                   -> "swap": 0
nameserver: 8.8.8.8       -> "dns": "8.8.8.8"

Compound fields need a small parser each:

net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,ip=192.168.1.200/24,type=veth
  -> "ipv4": "192.168.1.200/24"
  -> "gateway": "192.168.1.1"
  -> "bridge": "vmbr0"

rootfs: ssd:200/vm-200-disk-0.raw,size=200G
  -> "storage": "ssd"
  -> "disk": "200G"

features: keyctl=1,nesting=1
  -> "features": { "nesting": true, "keyctl": true }

mp0: /data/app/immich,mp=/mnt/immich
  -> "mount": { "source": "/data/app/immich", "target": "/mnt/immich" }

The unprivileged field needs inversion. Proxmox uses unprivileged: 1 to mean unprivileged. We store "privileged": false. If the key is missing, default is privileged (true).

The template field cannot be recovered from the conf. That info is lost after creation. We can use the current global template CTID if set, or just skip it.

[PoBruno]

There is a tension with the hostname convention.

Right now engine_resolve() always sets hostname to basename $PWD. That is the zero config magic: you create a folder called immich, run pve-compose up, and the container gets hostname immich automatically.

But with adopt, the container already exists with its own hostname. CT 201 has hostname owncloud and lives in /data/app/nextcloud. If adopt writes "hostname": "owncloud" to lxc.json, and then the user runs pve-compose plan, engine_resolve would overwrite it with nextcloud (basename of PWD).

The fix is straightforward. Make engine_resolve respect the hostname from lxc.json when it exists. Only fall back to basename when there is no lxc.json or the hostname field is missing.

# current behavior (always derives)
_hostname=$(basename "$(pwd)")

# new behavior (lxc.json wins if present)
_hostname=""
if [ -n "$_lxc_json" ]; then
    _hostname=$(printf "%s" "$_lxc_json" | jq -r ".hostname // empty")
fi
[ -n "$_hostname" ] || _hostname=$(basename "$(pwd)")

This is a small change but important. The convention becomes: basename is the default, lxc.json is the source of truth. Zero config still works exactly the same. Adopted containers keep their real hostname.

[PoBruno]

Thinking about the command interface.

Three options I considered.

Option A: explicit pve-compose adopt command. You cd into the directory, run adopt, it detects the container, generates lxc.json. Simple and clear. User knows what happened.

Option B: make pve-compose plan auto detect existing containers. If it finds a match, it generates lxc.json from the real config instead of defaults. Zero new commands. But it changes the meaning of plan from "show what will be created" to "show what exists or will be created". That can be surprising.

Option C: hybrid. adopt is the real command that generates lxc.json. plan just shows a hint when it detects an existing container. Something like "existing CT 200 matches this directory, run pve-compose adopt to import it". Plan keeps its original meaning, adopt is explicit.

Going with option C. Adopt should be an intentional action. It writes a file that changes how pve-compose treats that directory forever. That deserves an explicit command. But plan should be smart enough to tell you about it.

[PoBruno]

Sketching out what adopt looks like in practice.

cd /data/app/immich
pve-compose adopt
# Detected CT 200 (hostname: immich)
# Generated lxc.json from container config

cd /data/app/nextcloud
pve-compose adopt 201
# Adopted CT 201 (hostname: owncloud)
# Warning: hostname "owncloud" differs from directory name "nextcloud"
# Generated lxc.json

pve-compose adopt
# lxc.json already exists. Use pve-compose adopt 201 --force to overwrite.

The flow inside the command:

1. if lxc.json exists and no --force flag, stop
2. if CTID argument given, use it directly
3. if no CTID, scan conf files for mp0 source matching $PWD
4. if no mp0 match, try hostname matching as fallback
5. if still nothing, error "no container found for this directory"
6. read the conf file, extract all fields
7. build lxc.json with jq
8. if hostname != basename($PWD), print a warning
9. write lxc.json
10. done

After adopt, everything works. pve-compose ps, doctor, status, up, all of it. The container was already running, adopt just creates the metadata file that pve-compose needs.

[PoBruno]

This connects nicely with the pve-docker work planned in M13.

Both adopt and pve-docker need the same thing: read a conf file, extract fields, resolve a container by hostname or path. The parsing logic can live in a shared lib/resolve.sh that both features use.

Functions needed in resolve.sh:

resolve_ct_by_path(DIR)         find CTID by matching mp0 source path
resolve_ct_by_hostname(NAME)    find CTID by grepping hostname in conf files
resolve_conf_field(CTID, KEY)   read a simple field from the conf
resolve_net0_field(CTID, KEY)   parse compound net0 line, extract ip/gw/bridge
resolve_rootfs_field(CTID, KEY) parse rootfs line, extract storage/size
resolve_features(CTID)          parse features line, return nesting/keyctl bools
resolve_mp_field(CTID, N)       parse mpN line, return source and target
resolve_privileged(CTID)        read unprivileged flag and invert it

pve-docker uses resolve_ct_by_hostname and resolve_mp_field to route commands to the right container. adopt uses resolve_ct_by_path and all the field extractors to build the full lxc.json. The apply command could also reuse the net0 and features parsers instead of having its own copies.

One library, three consumers. That is the right level of reuse.

[PoBruno]

Implementation order.

Task 1: build lib/resolve.sh with all the conf parsing functions. No dependencies, can start right away. This is the foundation for everything else. Also serves M13 pve-docker later.

Task 2: build commands/adopt.sh. Depends on resolve.sh. The actual adopt command with auto detection, CTID argument, force flag, dry run flag.

Task 3: add detection hint to commands/plan.sh. When plan runs without lxc.json, check if a container matches the current directory. If yes, print a message suggesting adopt.

Task 4: modify engine_resolve() in lib/engine.sh to respect hostname from lxc.json. Small change, no dependencies, but needed for adopted containers where hostname differs from directory name.

Task 5: test everything on the real host. All 9 containers without lxc.json are test cases. The edge cases (owncloud/nextcloud mismatch, jellyfin on different storage, casa360 nested paths) are the important ones.

Task 1 (resolve.sh) ---+--- Task 2 (adopt.sh) ---+
                       +--- Task 3 (plan hint)    +--- Task 5 (E2E)
Task 4 (hostname fix) ----------------------------+

[PoBruno]

New ideia: Implementing native support to taggeds VLans
Thinking about..