Exosphere does not support ssh_config(5) Includes

[crpb]

Describe the bug
Included configurations from ssh_config are not being honored which doesn't allow for more complex ssh-configurations.
I use this to be able to share certain ssh-configurations for multiple customers with coworkers through an git-repo. This allows additions/changes/removals for hosts by any person that has the respective access. This is done by e.g. Include %d/git/work/ssh_config_support/*.conf which consists of configurations for each customer, test-systems, defaults, etc..
Managing all hosts in a single file or just /etc/ssh_config and ~/.ssh/config would end up in a disaster and people also getting configurations for various things like completion in your shell for hosts they don't even have access to as that also allows the use of multiple repos or just multiple configurations in your ~/.ssh/ or some other path for personal settings like my own VPS or systems at home.

To Reproduce

1. Create a "passwordless" pubkey that doesn't match the defaults (e.g. ssh -F /etc/ssh/ssh_config -G localhost |grep identity) and place it in authorized_keys.

This will just make sure that those defaults will not match by chance. You could also create a pubkey with an password and add it to your ssh-agent but it doesn't make a difference.

$ mkdir -m 0700 -p ~/.ssh
$ ssh-keygen -t ed25519 -P "" -f ~/.ssh/localhosttest
$ cat ~/.ssh/localhosttest.pub >> ~/.ssh/authorized_keys

2. Configure ssh_config with Include.

$ echo 'Include %d/.ssh/include' >> ~/.ssh/config
$ echo -e 'Host localhost\nIdentityFile %d/.ssh/localhosttest' >> ~/.ssh/config/include

3. Test SSH Connection

$ ssh -o StrictHostKeyChecking=accept-new -vvv localhost echo moo |& grep -e 'moo' -e 'identity' -e 'include' -e 'close'
debug3: Started with: ssh -vvv localhost echo moo
debug3: /home/test/.ssh/config line 1: Including file /home/test/.ssh/include depth 0
debug1: Reading configuration data /home/test/.ssh/include
debug1: /home/test/.ssh/include line 1: Applying options for localhost
debug1: identity file /home/test/.ssh/localhosttest type 2
debug1: no identity pubkey loaded from /home/test/.ssh/localhosttest
debug1: Sending command: echo moo
moo
debug2: channel 0: output drain -> closed
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: send_close2
debug2: channel 0: send close for remote id 0

4. Create exosphere configuration

$ mkdir -p ~/.config/exosphere
$ cat << EOF > ~/.config/exosphere/config.toml
[options]
debug = true
log_level = "DEBUG"

[[hosts]]
name = "localhost"
ip = "localhost"
EOF

5. Run a disvoery

test@obelix:~$ exosphere host discover localhost
2026-07-10 03:23:33,357 - exosphere.main - INFO - Logging initialized with level: DEBUG
2026-07-10 03:23:33,357 - exosphere.main - WARNING - Debug mode enabled! Logs may flood console!
2026-07-10 03:23:33,358 - exosphere.inventory - DEBUG - Initializing inventory with 1 hosts
2026-07-10 03:23:33,358 - exosphere.inventory - DEBUG - Loading state from cache file /home/test/.local/state/exosphere/exosphere.db
2026-07-10 03:23:33,358 - exosphere.inventory - DEBUG - Current Schema Version: 2
2026-07-10 03:23:33,360 - exosphere.inventory - DEBUG - Host localhost not found in cache, creating new
2026-07-10 03:23:33,361 - exosphere.pipelining - DEBUG - SSH pipelining disabled, not starting reaper thread
⠋ Discovering platform for 'localhost' 0:00:002026-07-10 03:23:33,366 - exosphere.objects - DEBUG - Pinging host localhost at localhost:22
2026-07-10 03:23:33,367 - exosphere.objects - DEBUG - Creating new connection to localhost using localhost:22, (timeout: 10)
2026-07-10 03:23:33,438 - exosphere.objects - ERROR - Ping to host localhost failed: No authentication methods available
2026-07-10 03:23:33,438 - exosphere.objects - DEBUG - Ping failed, will attempt platform detection for better error diagnosis
⠙ Discovering platform for 'localhost' 0:00:002026-07-10 03:23:33,503 - exosphere.setup.detect - ERROR - Unexpected error during OS detection: No authentication methods available
2026-07-10 03:23:33,504 - exosphere.objects - WARNING - Host localhost is offline, skipping discover.
2026-07-10 03:23:33,504 - exosphere.objects - DEBUG - Closed SSH connection to localhost (localhost:22)
╭─ Error ───────────────────────────────────────────╮
│ SSHException: No authentication methods available │
╰───────────────────────────────────────────────────╯
⠙ Discovering platform for 'localhost' 0:00:00
2026-07-10 03:23:33,506 - exosphere.main - DEBUG - Running exit handler for cache lock release
2026-07-10 03:23:33,506 - exosphere.main - DEBUG - Releasing cache lock on exit
2026-07-10 03:23:33,506 - exosphere.main - DEBUG - Running exit handler for connections cleanup
2026-07-10 03:23:33,506 - exosphere.main - DEBUG - Closing all SSH connections on exit
2026-07-10 03:23:33,506 - exosphere.inventory - DEBUG - Closing all SSH connections for 1 hosts
2026-07-10 03:23:33,506 - exosphere.objects - DEBUG - Closed SSH connection to localhost (localhost:22)
2026-07-10 03:23:33,506 - exosphere.objects - DEBUG - Clearing connection object for host localhost

Expected behavior
A successful discovery.

Where Exosphere runs:

• Platform: Debian Forky/Sid
• Exosphere Version: 3.0.0

Reference

• ssh_config(5) Include

[mrdaemon]

As far as I can reproduce programmatically, the issue stems from the Paramiko library (which sits underneath Fabric/Invoke) straight up not parsing Include statements within OpenSSH config files. It just gets ignored as a string literal at runtime.

There apparently have been many PRs and efforts over the years to get this done, at the last PR to propose this has been more or less sitting unmerged since 2024, which isn't ideal

paramiko/paramiko#2307

Given how Fabric works, I think I could work around it by adding a basic pre-loader that would resolve paths specified through 'Include' and merge them into the SSHConfig parameters at runtime -- we already do similar hackery for other reasons, so it's not too far fetched to imagine.

The only question is essentially how deep I'd be willing to go to implement this outside of the library: Include Semantics are more complex than just "also load this file in", since they can have nested includes, %d/%h/%u tokens, system vs user relative paths resolution etc.

Would you mind providing me with a more exhaustive example of your use case (anonymized, of course), in terms of what the configs look like? It would help me gauge how much I can do from my end. I'd very much like to support at least SOME of it here, as that's a fairly legitimate use case IMO, and I understand the need for it, outside of exosphere host config options, given it's likely shared with ssh access in general.

[mrdaemon]

Additionally: I've tagged this with documentation because either way, it needs to be documented.

[crpb]

So i went ahead to read a bit on paramiko and what keywords would even be supported and already began to think.. OH NOES :P.

Then i began to write configurations which still won't use Include and already ran into problems.

I put this here for now as it still shows how my included files look like (esp. the problematic part) but i'm guessing there should be another issue for it aswell (if there isn't one already).

Works

Exosphere Config

...
[[hosts]]
name = "localhost"
ip = "localhost"
[[hosts]]
name = "test2.host1"
ip = "test2.host1"
[[hosts]]
name = "test2.host2"
ip = "test2.host2"
[[hosts]]
name = "test2.host3"
ip = "test2.host3"

This already seemed a bit redundant if "name" == "ip(hostalias/hostname)". At least i can just use the hostaliases in ansible w/o any further configuration.

SSH-Config

1. Outgoing ProxyJump

# above is e.g. localhost
...
Host myproxy
	# Pubkey for user@hostalias.ed25519
	IdentityFile %d/.ssh/%r@%k.ed25519
	Hostname myproxy.example.com
	# Might have a dedicated jumpuser
	# User proxyuser

2. General Settings for target network

Match Host test2.*
	# One pubkey for all hosts in said network / user
	IdentityFile %d/.ssh/%r@test2.ed25519
	# Might have different/shared useraccount
	#User ...

3. Ingoing ProxyJump

Host test2.host1
	Hostname wanip.test2.example
	ProxyJump myproxy

4. Other Hosts

Host test2.host2
	Hostname 10.11.12.2
	ProxyJump test2.host1
Host test2.host3
	Hostname 10.11.12.3
	ProxyJump test2.host1
...

Problems

Now these last ProxyJump directives seem redundant right? And usually i'm not doing that this way.

I could remove those ProxyJump's from 4. Other Hosts and instead do the following before i declare them which will work with ssh as we not yet have set a ProxyJump for the matching network but only for test2.host1 which can't be overridden at a later point while during ssh parses all configurations(Include included :>). "First come, First Serve" iirc.

Match Host test2.*
	ProxyJump test2.host1

Another case would be the following to not use a proxyjump at all if we are on-site with a laptop and won't or might even can't use the proxyjumps at all.

Host gateway.example.com
	User foo
	Port 12345
# This will match if we don't have a ip from that network.
# Some use "Host" instead of "OriginalHost"
Match OriginalHost *.localdomain.example.com !Exec "ip a | grep ' 10.20.30.'"
	ProxyJump gateway.example.com

# Some host that can't be resolved by gateway.example.com
Host host1.localdomain.example.com
Hostname 10.20.30.2
# Hostdeclarations to make them known
Host host2.localdomain.example.com
Host host3.localdomain.example.com
...

Either case will break exosphere completly at least from what i could see here so far.

After further tinkering i found that a second Match with a ProxyJump will even break connecting to localhost for whatever reason. If i just set something like AddressFamily or User it still works.

Shortened Example

Host localhost
	...
Host myproxy
	...
Match test2.*
	IdentityFile ...
Host test2.host1
	ProxyJump myproxy
	Hostname ....
Match test2.*
	ProxyJump test2.host1
Host test2.host2
	Hostname ..
	ProxyJump test2.host1
Host test2.host3
	Hostname ..
	...
...

Logs

test@obelix:~$ exosphere inventory discover
2026-07-11 02:45:25,882 - exosphere.main - INFO - Logging initialized with level: DEBUG
2026-07-11 02:45:25,882 - exosphere.main - WARNING - Debug mode enabled! Logs may flood console!
2026-07-11 02:45:25,882 - exosphere.inventory - DEBUG - Initializing inventory with 4 hosts
2026-07-11 02:45:25,882 - exosphere.inventory - DEBUG - Loading state from cache file /home/test/.local/state/exosphere/exosphere.db
2026-07-11 02:45:25,882 - exosphere.inventory - DEBUG - Current Schema Version: 2
2026-07-11 02:45:25,885 - exosphere.inventory - DEBUG - Loading host state for localhost from cache
2026-07-11 02:45:25,885 - exosphere.inventory - DEBUG - Applying cached state to host localhost
2026-07-11 02:45:25,885 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:45:25,885 - exosphere.inventory - DEBUG - Loading host state for host1 from cache
2026-07-11 02:45:25,886 - exosphere.inventory - DEBUG - Applying cached state to host host1
2026-07-11 02:45:25,886 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:45:25,886 - exosphere.inventory - DEBUG - Loading host state for host2 from cache
2026-07-11 02:45:25,886 - exosphere.inventory - DEBUG - Applying cached state to host host2
2026-07-11 02:45:25,886 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:45:25,886 - exosphere.inventory - DEBUG - Loading host state for host3 from cache
2026-07-11 02:45:25,886 - exosphere.inventory - DEBUG - Applying cached state to host host3
2026-07-11 02:45:25,886 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:45:25,886 - exosphere.pipelining - DEBUG - SSH pipelining disabled, not starting reaper thread
2026-07-11 02:45:25,888 - exosphere.commands.inventory - INFO - Gathering platform information for hosts
Gathering platform information ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--2026-07-11 02:45:25,891 - exosphere.inventory - DEBUG - Dispatching discover to 4 host(s)
2026-07-11 02:45:25,891 - exosphere.inventory - DEBUG - Using ThreadPoolExecutor with 5 threads
2026-07-11 02:45:25,891 - exosphere.inventory - DEBUG - Submitting 4 tasks to executor for method 'discover'
2026-07-11 02:45:25,891 - exosphere.objects - DEBUG - Pinging host localhost at localhost:22
2026-07-11 02:45:25,893 - exosphere.objects - DEBUG - Pinging host host1 at test2.host1:22
2026-07-11 02:45:25,895 - exosphere.objects - DEBUG - Pinging host host2 at test2.host2:22
2026-07-11 02:45:25,897 - exosphere.objects - DEBUG - Pinging host host3 at test2.host3:22
2026-07-11 02:45:25,897 - exosphere.objects - DEBUG - Creating new connection to host1 using test2.host1:22, (timeout: 10)
2026-07-11 02:45:25,897 - exosphere.objects - DEBUG - Creating new connection to localhost using localhost:22, (timeout: 10)
2026-07-11 02:45:25,900 - exosphere.objects - DEBUG - Creating new connection to host2 using test2.host2:22, (timeout: 10)
2026-07-11 02:45:25,900 - exosphere.objects - DEBUG - Creating new connection to host3 using test2.host3:22, (timeout: 10)
Gathering platform information ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--2026-07-11 02:45:27,019 - exosphere.objects - ERROR - Ping to host host2 failed: maximum recursion depth exceeded
2026-07-11 02:45:27,023 - exosphere.objects - DEBUG - Ping failed, will attempt platform detection for better error diagnosis
2026-07-11 02:45:27,024 - exosphere.objects - ERROR - Ping to host host1 failed: maximum recursion depth exceeded
2026-07-11 02:45:27,027 - exosphere.objects - DEBUG - Ping failed, will attempt platform detection for better error diagnosis
2026-07-11 02:45:27,029 - exosphere.objects - DEBUG - Creating new connection to host1 using test2.host1:22, (timeout: 10)
2026-07-11 02:45:27,032 - exosphere.objects - DEBUG - Creating new connection to host2 using test2.host2:22, (timeout: 10)
2026-07-11 02:45:27,074 - exosphere.objects - ERROR - Ping to host host3 failed: maximum recursion depth exceeded
2026-07-11 02:45:27,074 - exosphere.objects - DEBUG - Ping failed, will attempt platform detection for better error diagnosis
2026-07-11 02:45:27,087 - exosphere.objects - DEBUG - Creating new connection to host3 using test2.host3:22, (timeout: 10)
Gathering platform information ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--2026-07-11 02:45:27,095 - exosphere.objects - ERROR - Ping to host localhost failed: maximum recursion depth exceeded
2026-07-11 02:45:27,096 - exosphere.objects - DEBUG - Ping failed, will attempt platform detection for better error diagnosis
2026-07-11 02:45:27,102 - exosphere.objects - DEBUG - Creating new connection to localhost using localhost:22, (timeout: 10)
Gathering platform information ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--2026-07-11 02:45:27,997 - exosphere.inventory - ERROR - Failed to run discover on host1: maximum recursion depth exceeded
[FAILED] host1
Gathering platform information ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  25% -:--:--2026-07-11 02:45:28,055 - exosphere.inventory - ERROR - Failed to run discover on host2: maximum recursion depth exceeded
[FAILED] host2
Gathering platform information ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  25% -:--:--2026-07-11 02:45:28,087 - exosphere.inventory - ERROR - Failed to run discover on localhost: maximum recursion depth exceeded
[FAILED] localhost
Gathering platform information ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  25% -:--:--2026-07-11 02:45:28,095 - exosphere.inventory - ERROR - Failed to run discover on host3: maximum recursion depth exceeded
[FAILED] host3

The following hosts could not be discovered due to errors:

  host1       maximum recursion depth exceeded
  host2       maximum recursion depth exceeded
  localhost   maximum recursion depth exceeded
  host3       maximum recursion depth exceeded

2026-07-11 02:45:28,102 - exosphere.commands.utils - DEBUG - Starting inventory save operation
⠋ Saving inventory state to disk  0:00:002026-07-11 02:45:28,104 - exosphere.inventory - INFO - Saving inventory state to cache file /home/test/.local/state/exosphere/exosphere.db
2026-07-11 02:45:28,153 - exosphere.commands.utils - DEBUG - Inventory save operation completed
2026-07-11 02:45:28,153 - exosphere.main - DEBUG - Running exit handler for cache lock release
2026-07-11 02:45:28,153 - exosphere.main - DEBUG - Releasing cache lock on exit
2026-07-11 02:45:28,153 - exosphere.main - DEBUG - Running exit handler for connections cleanup
2026-07-11 02:45:28,153 - exosphere.main - DEBUG - Closing all SSH connections on exit
2026-07-11 02:45:28,153 - exosphere.inventory - DEBUG - Closing all SSH connections for 4 hosts

test@obelix:~$ exosphere inventory refresh
2026-07-11 02:56:00,087 - exosphere.main - INFO - Logging initialized with level: DEBUG
2026-07-11 02:56:00,087 - exosphere.main - WARNING - Debug mode enabled! Logs may flood console!
2026-07-11 02:56:00,088 - exosphere.inventory - DEBUG - Initializing inventory with 4 hosts
2026-07-11 02:56:00,088 - exosphere.inventory - DEBUG - Loading state from cache file /home/test/.local/state/exosphere/exosphere.db
2026-07-11 02:56:00,088 - exosphere.inventory - DEBUG - Current Schema Version: 2
2026-07-11 02:56:00,091 - exosphere.inventory - DEBUG - Loading host state for localhost from cache
2026-07-11 02:56:00,091 - exosphere.inventory - DEBUG - Applying cached state to host localhost
2026-07-11 02:56:00,091 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:56:00,091 - exosphere.inventory - DEBUG - Loading host state for host1 from cache
2026-07-11 02:56:00,091 - exosphere.inventory - DEBUG - Applying cached state to host host1
2026-07-11 02:56:00,091 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:56:00,091 - exosphere.inventory - DEBUG - Loading host state for host2 from cache
2026-07-11 02:56:00,091 - exosphere.inventory - DEBUG - Applying cached state to host host2
2026-07-11 02:56:00,091 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:56:00,091 - exosphere.inventory - DEBUG - Loading host state for host3 from cache
2026-07-11 02:56:00,092 - exosphere.inventory - DEBUG - Applying cached state to host host3
2026-07-11 02:56:00,092 - exosphere.providers.apt - DEBUG - Initializing Debian Apt package manager
2026-07-11 02:56:00,092 - exosphere.pipelining - DEBUG - SSH pipelining disabled, not starting reaper thread
2026-07-11 02:56:00,096 - exosphere.commands.inventory - INFO - Refreshing inventory data
Refreshing package updates ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--2026-07-11 02:56:00,098 - exosphere.inventory - DEBUG - Dispatching refresh_updates to 4 host(s)
2026-07-11 02:56:00,098 - exosphere.inventory - DEBUG - Using ThreadPoolExecutor with 5 threads
2026-07-11 02:56:00,098 - exosphere.inventory - DEBUG - Submitting 4 tasks to executor for method 'refresh_updates'
2026-07-11 02:56:00,099 - exosphere.inventory - ERROR - Failed to run refresh_updates on host1: Host host1 is offline.
2026-07-11 02:56:00,099 - exosphere.inventory - ERROR - Failed to run refresh_updates on localhost: Host localhost is offline.
2026-07-11 02:56:00,100 - exosphere.inventory - ERROR - Failed to run refresh_updates on host2: Host host2 is offline.
2026-07-11 02:56:00,100 - exosphere.inventory - ERROR - Failed to run refresh_updates on host3: Host host3 is offline.
2026-07-11 02:56:00,101 - exosphere.commands.utils - DEBUG - Starting inventory save operation
⠋ Saving inventory state to disk  0:00:002026-07-11 02:56:00,103 - exosphere.inventory - INFO - Saving inventory state to cache file /home/test/.local/state/exosphere/exosphere.db
2026-07-11 02:56:00,149 - exosphere.commands.utils - DEBUG - Inventory save operation completed

              Refresh Errors

  host1       Host host1 is offline.
  localhost   Host localhost is offline.
  host2       Host host2 is offline.
  host3       Host host3 is offline.

2026-07-11 02:56:00,151 - exosphere.main - DEBUG - Running exit handler for cache lock release
2026-07-11 02:56:00,151 - exosphere.main - DEBUG - Releasing cache lock on exit
2026-07-11 02:56:00,151 - exosphere.main - DEBUG - Running exit handler for connections cleanup
2026-07-11 02:56:00,151 - exosphere.main - DEBUG - Closing all SSH connections on exit
2026-07-11 02:56:00,151 - exosphere.inventory - DEBUG - Closing all SSH connections for 4 hosts

And now we haven't even yet Included anything ^_^.

FWIW, my includes are only one level deep (so no nested Match Includes!) and make use of either ~/path/to/files/*.conf (extensions might vary so this would need to be looked at with os.glob iirc) or instead the tilde use the token %d, no other tokes are used here (for now?). The latter i learned about at a later time which is the reason this is mixed but evidently paramiko at least wouldn't have a problem with it as it looks. I guess this only matters for SSHConfig if you try to replace those tokens on matched '^Include ... lines to load those paths manually.

My folderstructure in those directories will kinda look like this. (we remember first come first serve :>)

000_defaults.conf
001_something.conf
002...conf
005_work.conf
010_cust1.conf
010_cust2.conf
020_test.conf
...

[tobiasstein]

From my humble perspective Include directives are basically working, because included hosts directly reachable via ssh are discovered fine by exosphere.
A ProxyJump on an included host seems not to be honored.
If the same host is defined on top-level ~/.ssh/config, the ProxyJump is honored and the exosphere discovery works as expected.