Sonoma 14.6 breaks ssh-askpass

[lukaskuzmiak]

Just after update to Sonoma 14.6 today I get:

sign_and_send_pubkey: signing failed for ED25519 "/Users/lukash/.ssh/<redacted>" from agent: agent refused operation

Same for my co-worker, it definitely worked ok on 14.5 just before this update (hours ago).

[dataviruset]

Seems the service can't start:

√ ~$ brew services restart theseal/ssh-askpass/ssh-askpass
Stopping `ssh-askpass`... (might take a while)
==> Successfully stopped `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)
==> Successfully started `ssh-askpass` (label: homebrew.mxcl.ssh-askpass)

But it's still not running:

√ ~$ brew services info ssh-askpass
ssh-askpass (homebrew.mxcl.ssh-askpass)
Running: ✘
Loaded: ✔
Schedulable: ✘

When trying to start it:

√ ~$ brew services start ssh-askpass
Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/redacted/Library/LaunchAgents/homebrew.mxcl.ssh-askpass.plist` exited with 5.
?1 ~$ sudo brew services start ssh-askpass
Password:
Warning: Taking root:admin ownership of some ssh-askpass paths:
  /bin
  /bin/sh
  /opt/homebrew/opt/ssh-askpass
  /opt/homebrew/opt/ssh-askpass/bin
  /opt/homebrew/var/homebrew/linked/ssh-askpass
This will require manual removal of these paths using `sudo rm` on
brew upgrade/reinstall/uninstall.
Error: Operation not permitted @ apply2files - /bin

I tried deleting the /opt/homebrew folders mentioned and reinstalling ssh-askpass but it didn't help. I don't wanna try what happens if I delete /bin or /bin/sh... 😅
The owner of /bin is root:wheel these days and not root:admin.

[zeyugao]

@dataviruset You can directly set SSH_ASKPASS and SUDO_ASKPASS to test it without the services. It is due to the system I think, not the services

export SSH_ASKPASS=/opt/homebrew/opt/touch-auth/bin/touch-auth  # Change it
unset SSH_AUTH_SOCK
echo $SSH_AUTH_SOCK
eval $(ssh-agent -s)
echo $SSH_AUTH_SOCK

[dataviruset]

@dataviruset You can directly set SSH_ASKPASS and SUDO_ASKPASS to test it without the services. It is due to the system I think, not the services

export SSH_ASKPASS=/opt/homebrew/opt/touch-auth/bin/touch-auth  # Change it
unset SSH_AUTH_SOCK
echo $SSH_AUTH_SOCK
eval $(ssh-agent -s)
echo $SSH_AUTH_SOCK

Using my original SSH_ASKPASS value (it's set to /opt/homebrew/opt/ssh-askpass/bin/ssh-askpass) and then running these commands worked for me:

unset SSH_AUTH_SOCK
eval $(ssh-agent -s)
ssh-add -c ~/.ssh/my_key
ssh test-machine.example.com

Interestingly the SSH_ASKPASS environment variable is already set. Probably the brew package set that up for me. But the built-in Mac OS SSH agent seems to not work anymore as of Mac OS 14.6...

[EXHades]

I added keys to the ssh-agent using keepassxc.
just deselect Require user confirmation when this key is used option, locked keepassxc and then unlocked it again, Now ssh -T example.com works fine, But SSH_ASKPASS still doesn't work.

keepassxreboot/keepassxc#9955 (comment)

---

macOS 14.6 upgrade break ssh-agent SSH_AGENT_CONSTRAIN_CONFIRM support?

[micolous]

If your organisation depends on this working and has a support agreement with Apple, please point them at this bug. 😄

I dug in to this a bit today, it looks like Sonoma 14.6 has changed the environment variable inheritance rules for system-provided user daemons (/System/Library/LaunchAgents); so that environment variables set with launchctl setenv no longer apply.

This means that when macOS starts ssh-agent through launchctl (to handle the default SSH_AUTH_SOCK on demand), it no longer picks up SSH_ASKPASS and DISPLAY from this ssh-agent's launchd plist.

The work-around @zeyugao proposed side-steps all of this by not using macOS' ssh-agent config – but this only works in a per-shell context. Similarly, if you replaced ssh with the one from Homebrew and put it first in your path; or used a different environment variable to SSH_AUTH_SOCK to point at a custom ssh-agent, you'd also work around this... but that also kinda smells.

On a system with Sonoma 14.5 with xquartz installed, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		DISPLAY => /private/tmp/com.apple.launchd.XXXX/org.macosforge.xquartz:0
		SUDO_ASKPASS => /usr/local/opt/ssh-askpass/bin/ssh-askpass
		SSH_ASKPASS => /usr/local/opt/ssh-askpass/bin/ssh-askpass
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

On a system with Sonoma 14.6 without xquartz, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

With Sonoma 14.6 with xquartz installed, I see:

% launchctl print gui/501/com.openssh.ssh-agent
gui/501/com.openssh.ssh-agent = {
	active count = 1
	path = /System/Library/LaunchAgents/com.openssh.ssh-agent.plist
	type = LaunchAgent
	state = running

	program = /usr/bin/ssh-agent
	arguments = {
		/usr/bin/ssh-agent
		-l
	}

	inherited environment = {
		DISPLAY => /private/tmp/com.apple.launchd.XXXX/org.xquartz:0
		SSH_AUTH_SOCK => /private/tmp/com.apple.launchd.XXXX/Listeners
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		MallocSpaceEfficient => 1
		XPC_SERVICE_NAME => com.openssh.ssh-agent
	}

Other macOS system daemons (eg: com.apple.weatherd) are impacted in the same way – so this is not specific to ssh-agent or the way ssh-askpass works.

User-installed software in /Library/LaunchAgents and ~/Library/LaunchAgents don't seem to be affected by this change.

The fact that xquartz (a non-system application) can still set the DISPLAY environment variable for ssh-agent suggests that it should be possible for this ssh-askpass to set environment variables in some way, but from what I'm reading of the launchd.plist man page, it won't be elegant.

I suspect that this was part of some other security / platform hardening change, but I'm unable to find any specific release notes about this.

It's something that Apple will need to fix.

[zeyugao]

@micolous Have you succeeded in setting SSH_ASKPASS to workaround this manually? I just propose a method to bypass the usage of services as @dataviruset says the services failed to start.

I manually set the SSH_ASKPASS and started a ssh-agent, but I still got agent refused operation

[micolous]

@micolous Have you succeeded in setting SSH_ASKPASS to workaround this manually?

No.

What matters is the environment that ssh-agent runs in, which when it's spawned by launchd, ignores launchctl setenv on macOS Sonoma 14.6.

The error we're seeing is because ssh-agent has no way to ask us for confirmation:

• if DISPLAY is set, SSH_ASKPASS is still unset, so it doesn't know what to do and reject it.
• if DISPLAY is unset, then it'll ask on stdin... but that's not actually connected to anything so it'll reject it.

The work-around I've got for now is to disable confirmation, but I've set up my SSH keys using /usr/lib/ssh-keychain.dylib to access my keys over PKCS#11 / smartcard auth, and the smart card (Yubikey with PIV applet) is configured to require physical confirmation for signing operations.

It just means now I don't get any prompt on screen, just a flashing light on the side of my computer.

If you're using ssh-askpass to confirm access to a key stored on disk, then disabling confirmation has a pretty significant security drawback 😓

[lukaskuzmiak]

I worked around it by running a user-land instance of ssh-agent with proper environment variables set, then overwriting SSH_AUTH_SOCK in .zprofile and on top of that setting IdentityAgent in ~/.ssh/config to the path of the user-land ssh-agent.

Some GUI apps might not pick that up, but so far without issues. Other than it being annoying to setup.

For anyone interested here are some resources that helped me:

• https://gist.github.com/vancluever/de1c3985c8f9e2a3c4bdc42a057e075e
• 1Password uses a similar approach as described above and has a nice client compatibility table here: https://developer.1password.com/docs/ssh/agent

Given the fact Apple likely broke this for good I think that will be the way forward.

[micolous]

Yeah, your setup and 1Password's replace the default Apple-provided SSH agent setup (partially or entirely). There are usability edge cases to that, as you've already noted. If you go further (as 1Password do, and has been suggested elsewhere) and replace ssh-agent entirely, you also miss out on security hardening features only available in Apple's SSH binaries.

I acknowledge that there are tradeoffs, like that those hardening features can also prevent you from using some more "obscure" functionality (like FIDO2 resident keys and PIV with ECDSA keys).

For what it's worth, there are large corporates and government agencies with macOS workstations and a custom ssh-askpass implementation (to support PKCS#11 auth with hardware keystores), but their corporate IT teams have probably noticed this in testing and are holding back the update from their fleets.

At this point, I'm quietly optimistic that it won't remain broken forever... it just needs to get in front of the right set of eyes. 😄

[sprig]

I think the issue has been (at lease partially) misdiagnosed by @micolous;

(FWIW I use a different askpass program but I think the issue is similar)
First, I've noticed that if I don't initially load keys into ssh-agent then I do get asked for a password by ssh-askpass. However if I do load keys ssh complains that agent refused operation.

As an experiment, I tried unsetting the SSH_ASKPASS variable in the terminal but setting SSH_ASKPASS_REQUIRE=force, and running ssh -vv host:
I notice lines like the following:

ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory

So I add a tiny script to the above location:

#!/bin/sh
echo "$@" > ~/askpass
date >> ~/askpass
env >> ~/askpass

I notice that when keys are not loaded, I get output as follows:

$ cat ~/askpass
Enter passphrase for key '/Users/user/.ssh/id_rsa': 
Sun Aug  4 11:07:55 PDT 2024
TERM_PROGRAM=Apple_Terminal
LC_MONETARY=en_US.UTF-8
QT_STYLE_OVERRIDE=Fusion
LUA_BINDIR=<redacted>
TERM=xterm-256color
SHELL=/bin/zsh
TMPDIR=/var/folders/vx/wq1n25gs5t54v4sp3_6yj3c00000gn/T/
TERM_PROGRAM_VERSION=453
LC_NUMERIC=en_US.UTF-8
CURL_SSL_BACKEND=secure-transport
TERM_SESSION_ID=516E43F7-2240-481C-A308-A13AD18FE7B4
LC_ALL=en_US.UTF-8
CDPATH=.:<redacted>
USER=user
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.LVjkwSsCS6/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PAGER=less
PATH=<long list of paths>
LC_MESSAGES=en_US.UTF-8
LUA_DIR=<redacted>
LaunchInstanceID=9A548B16-A27C-4770-B569-8A81CC12830F
QT_QPA_PLATFORMTHEME=gtk2
WORDCHARS=
__CFBundleIdentifier=com.apple.Terminal
LC_COLLATE=en_US.UTF-8
PROFILE_RAN=Sun Aug  4 09:49:08 PDT 2024
PWD=/Users/user
EDITOR=vim
LANG=en_US.UTF-8
LUA_PATH=<redacted>
XPC_FLAGS=0x0
SSH_ASKPASS_REQUIRE=force
XPC_SERVICE_NAME=0
SHLVL=2
HOME=/Users/user
LOGNAME=user
PYTHONPATH=<redacted>
LESS=-F -g -i -M -R -S -w -z-4
LC_CTYPE=en_US.UTF-8
BROWSER=open
DISPLAY=/private/tmp/com.apple.launchd.NVQ3vVyfbz/org.xquartz:0
SECURITYSESSIONID=186ab
LC_TIME=en_US.UTF-8
_=/usr/bin/env

In particular, the PROFILE_RAN variable above is set to date whenever my ~/.profile is sourced, one can notice that it is significantly different from when the askpass script was run. In fact, it is identical to what I have in that particular terminal session:

$ echo $PROFILE_RAN
Sun Aug  4 09:49:08 PDT 2024

i.e. the script runs in the same context as the ssh command, likely invoked directly by ssh.

Whereas if keys are loaded I get similar to the following:

$ cat ~/askpass 
Allow use of key user@host?
Key fingerprint SHA256:<redacted>.
SHELL=/bin/zsh
SSH_ASKPASS_PROMPT=confirm
TMPDIR=/var/folders/vx/wq1n25gs5t54v4sp3_6yj3c00000gn/T/
MallocSpaceEfficient=1
USER=user
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.LVjkwSsCS6/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PWD=/
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/user
LOGNAME=user
DISPLAY=/private/tmp/com.apple.launchd.NVQ3vVyfbz/org.xquartz:0
_=/usr/bin/env

which is a fairly vanilla environment, likely invoked by ssh-agent. Furthermore, since the script above returns 0 (success), the authentication is successful.

The conclusion is that rather than not recognizing SSH_ASKPASS environment variable which is read by ssh and not by the agent, ssh-agent fails to create a window when invoked by ssh-agent. Likely similar issues to what prompted reattach-to-user-namespace in https://github.com/ChrisJohnsen/tmux-MacOSX-pasteboard. I don't know if that tool still works but if it doesn't this suggests that another viable workaround; Separate this into a client/server setup where the server would run in the user's session and be able to open windows whereas the client would simply forward authentication requests from the agent.

[sprig]

update: I can confirm that reattach-to-user-namespace still works and using a wrapper script in /usr/X11R6/bin/ssh-askpass such as

#!/bin/sh
exec ~/local/bin/reattach-to-user-namespace ~/local/ssh-askpass/ssh-askpass "$@"

where programs are at the respective locations above opens the confirmation dialog and proceeds to successfully authenticate. Presumably pointing SSH_ASKPASS at the wrapper would work as well.

EDIT:
Upon further testing it seems that the issue @micolous identified is at play here as well - the environment does not get set. However, the path above is the default location where ssh-askpass is searched for and so putting the wrapper there still works as a workaround.

[micolous]

I can confirm that reattach-to-user-namespace still works and using a wrapper script in /usr/X11R6/bin/ssh-askpass

I tried your work-around on Sonoma 14.5 and 14.6 with XQuartz installed (so DISPLAY is set by SecureSocketWithKey, rather than launchctl setenv).

I removed the ssh-askpass.plist LaunchAgent and cleared out any traces from it from my shell and reset ssh-agent back to an initial state with no loaded keys:

brew services stop ssh-askpass
launchctl unsetenv SSH_ASKPASS
launchctl unsetenv SUDO_ASKPASS
unset SSH_ASKPASS
unset SUDO_ASKPASS
killall ssh-agent

I then added the keys again with confirmation enabled:

# For SSH key in the Keychain:
ssh-add -cs /usr/lib/ssh-keychain.dylib
# If you were using a private key stored on disk, you would do:
ssh-add -c

I tested putting both your tiny script and this repository's version of ssh-askpass at /usr/X11R6/bin/ssh-askpass (ie: without reattach-to-user-namespace), and they both ran without issues on both Sonoma 14.5 and 14.6.

Even adding a symlink to a Homebrew-installed version of ssh-askpass worked fine.

Looking at the source of Apple's version of SSH, it will try to use /usr/X11R6/bin/ssh-askpass if the SSH_ASKPASS environment variable is unset:

https://github.com/apple-oss-distributions/OpenSSH/blob/9b6202341ee10b42e7391229ad5c0f2eb8aea8af/openssh/readpass.c#L174-L178

This would suggest that the only trigger for the issue is that environment variables from launchctl setenv are no longer propagated to system daemons, which means ssh-askpass must be at /usr/X11R6/bin/ssh-askpass.

While I don't doubt that reattach-to-user-namespace is useful in some contexts, it seems that at least for GUI sessions, ssh-agent runs in a namespace where child processes can create working dialogs. With this version of ssh-askpass, you can check the parent process for osascript in Activity Monitor and confirm where it's spawned from.

(This post was edited multiple times to clarify things.)

[micolous]

A step-by-step, minimum-viable workaround for Sonoma 14.6 and later with macOS' default, out-of-the-box ssh-agent setup, based on @sprig's comment and my previous comment:

1. Remove any LaunchAgents which attempt to replace Apple's ssh-agent, manipulate SSH_AUTH_SOCK or the file it points to.

2. Install ssh-askpass, but don't install ssh-askpass.plist to ~/Library/LaunchAgents/ or register it with launchctl (unless you want SUDO_ASKPASS).

   If the LaunchAgent is already installed (eg: via Homebrew), it technically doesn't matter – it just has no effect on ssh-agent on Sonoma 14.6 and later.

3. Install XQuartz. This can be downloaded from the project's website or can be installed with brew install xquartz.

   Note: You don't have to actually use or run XQuartz – this is just so that the DISPLAY environment variable is set using SecureSocketWithKey, which works for system LaunchAgents like Apple's ssh-agent.

4. Log out of macOS and log in again. This makes Apple's ssh-agent pick up XQuartz' new environment variables from launchd.

5. Check that the DISPLAY environment variable would be set for Apple's ssh-agent (under "inherited environment"):

   launchctl print gui/$UID/com.openssh.ssh-agent

   If DISPLAY isn't set, then XQuartz is not installed correctly (check your start-up items, and log out and log in again).

6. Terminate any running ssh-agent processes, which may have old environment variables:

   killall -v ssh-agent

7. Create a symlink to where you installed ssh-askpass in /private/var/select/X11/bin (which is in turn symlinked from /usr/X11R6/bin):

   # Create /private/var/select/X11/bin if it doesn't already exist
   sudo mkdir -p /private/var/select/X11/bin
   
   # Link to where you installed ssh-askpass at /private/var/select/X11/bin/ssh-askpass, eg:
   # If you installed ssh-askpass with Homebrew on Apple Silicon:
   sudo ln -s /opt/homebrew/bin/ssh-askpass /private/var/select/X11/bin/ssh-askpass
   # If you installed ssh-askpass with Homebrew on Intel:
   sudo ln -s /usr/local/bin/ssh-askpass /private/var/select/X11/bin/ssh-askpass

Then you can load your keys into your SSH agent as normal (ssh-add -c...)

This should also work on Sonoma 14.5 and earlier, if you want to prepare a system before upgrading to 14.6.

A work-around without XQuartz would be to have ssh-askpass.plist pretend to be an X server with using SecureSocketWithKey (like XQuartz's LaunchAgent does). But this would break any application that actually tried to use that X server (probably in surprising or difficult to debug ways), and conflict with XQuartz.

Edited (2024-08-06): Fix typo: replace /var/private with /private/var

Edited (2025-02-18): Added that you may need to log out and log in again after installing XQuartz for the DISPLAY environment variable to show up properly (tested with Homebrew package on Sequoia 15.3.1). More explicitly noted that setting environment variables with SecureSocketWithKey is what makes this all work. Noted Apple Silicon Homebrew install location for ssh-askpass.

[D54]

An alternative workaround is to clone and shadow the original system launch-agent, which cannot be disabled or modified (due to SIP), and does not inherit global environment.

Make a copy:

cp /System/Library/LaunchAgents/com.openssh.ssh-agent.plist ~/Library/LaunchAgents/com.openssh.ssh-agent-my.plist

Change:

• the Label to e.g. com.openssh.ssh-agent-my
• and the SecureSocketWithKey to e.g. SSH_AUTH_SOCK_MY

Load the launch-agent

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.openssh.ssh-agent-my.plist

Even if it's the same executable, it does inherit the global environment as it is not a system launch-agent. So it gets the $SSH_ASKPASS variable.

Restart your terminal app to acquire the fresh global env.

Now the $SSH_AUTH_SOCK_MY environment variable should contain a new launchctl listener socket-file.

To make it work in your terminal, you can override the original env variable in your shell config:

export SSH_AUTH_SOCK=$SSH_AUTH_SOCK_MY

To make it work with all apps, we have to override the socket-file itself:

ln -sf $SSH_AUTH_SOCK_MY $SSH_AUTH_SOCK

(Idea source: maxgoedjen/secretive

Finally, it's working just as before the OS upgrade.

[sprig]

@micolous - In my case ssh-askpass stopped working without using the reattach-to-user-namespace wrapper. Just putting the ssh-askpass script in the above location makes ssh fail without any dialog opening. EDIT: Actually you are correct - reattach-to-user-namespace is not required. ssh-askpass.plist is still useful if you ever try to ssh with a key that isn't loaded into ssh-agent (or using a password). I don't know regarding SUDO_ASKPASS - I never had any dialog popup when I use sudo, when this is set.

However, there's no mention of /var/private anywhere on my system so I'm not sure where you got the information regarding it being symlinked from /usr/X11R6/bin. In my case /usr/X11R6 is a symlink to /opt/x11, whereas /usr/X11 is linked to /private/var/select/X11, (/private/var, not /var/private!) which itself in turn is again linked to /opt/x11. I'm guessing that was a typo. Regardless, I'm not sure why you would need to create the directory if it already is a symlink.