Improve port management

[Lumycea]

Currently, the ports exposed are manually assigned, which is not very convenient. This includes ports exposed to the Internet through Caddy and those used to backup databases. We could try to setup a script to dynamically choose those ports, or even better find a way to setup overlay networks containing the host.

For the databases backups, we could go around this difficulty and move the backup script to a container, but I'm a bit reluctant since it would prevent backups if there is an issue with docker. What do you think about that ?

[mochiruntime]

@Thechi2000 IMO having ports listed explicitly as part of IaC makes it well-defined, reproducible and self-explanatory. I would understand if we had a more complex use case, where e.g. a service mesh would be useful, but otherwise I think it adds unnecessary complexity. For public ports, do we have something else than the standard 80, 443 and backup ports?

Do you have a use case in mind where the current process is too constraining?

[Lumycea]

For public ports, do we have something else than the standard 80, 443 and backup ports?

Yes, there are a few (SSH, webhook, old wiki, onlyoffice), but I was thinking rather about the ports exposed to Caddy and Borgmatic.

Do you have a use case in mind where the current process is too constraining?

I don't find it constraining, but when adding/removing services, it requires to find available ports, which is quite error-prone given our current workflow. I just though about it, but we could have a section in the documentation clearly labeling conventions (e.g. 8--- for caddy exposed services) and the used ports, as well as the holes in the assignation (when removing a service).

[mochiruntime]

don't find it constraining, but when adding/removing services, it requires to find available ports

@Thechi2000 ah right, that's because caddy is bare-metal for IPv6 right? IMHO just get rid of IPv6 support and stay docker-native, this removes a layer of attack surface and complexity that is not that useful. Though you may have a use case which I don't know of. In any case +1 for docs

[Lumycea]

Sadly, it's not that simple. The current DNS configuration, which we cannot modify, specifies both IPv4 and IPv6, thus any attempt to connect to the server will occur over IPv6. I was not able to find a way to cleanly reverse-proxy packets from IPv6 to IPv4, but I would be happy if you have a solution.

[NoeTerrier]

I think it's a good idea to have a dynamic assignment of ports inside the docker networks if it makes adding a service to the stack easier and without more complexity.

[Lumycea]

Due to the recent migration back to Lys, the server will most likely only be accessible through IPv4. This means that we could migrate the caddy instance to a docker stack and route everything with aliases in the docker networks.

This would work well in the context of the docker stacks, but we should investigate how to handle databases backup, as for now it relies on the database ports being exposed.

PS: Looking back at the IPv4/6 issue, perhaps we could try exposing the dockerized caddy instance on another port like 8080/8443 (thus only available through IPv4), then setup a tunnel between :80 to 0.0.0.0:80, which may solve the issue completely.