Timeout on RunspacePool

[1upbyte]

Using 1.0.0b1

After (roughly) two minutes of inactivity, the RunspacePool will no longer allow sending commands. Creating a new pipeline will still work, but pipeline.invoke() will crash. Furthermore, it hangs the Python program and remains connected on the Windows host for a while.

Here is my example script to reproduce:

test.py

import time
from psrp import WSManInfo, SyncRunspacePool
import psrp

hostname = "example.com"
username = "user"
password = "pass"

conn = WSManInfo(
    hostname,
    username=username,
    password=password,
)

with SyncRunspacePool(conn) as rp:
    print("Starting runspace pool...")
    print(rp._pool.runspace_pool_id)
    ps = psrp.SyncPowerShell(rp)
    output = ps.add_script("date").add_command("Out-String").invoke()[0].strip()
    print(output)
    sleep_time = 130
    time.sleep(sleep_time)
    print(f"Slept {sleep_time}")
    ps = psrp.SyncPowerShell(rp)
    output = ps.add_script("date").add_command("Out-String").invoke()
    print(output[0])

Even after the script ends (hangs or is forcibly killed), the state of the runspace still shows as connected on Windows:

PS C:\Users\Administrator>  Get-WSManInstance -ConnectionURI 'http://localhost:5985/wsman' -ResourceURI shell -Enumerate  | Format-Table ShellId,ClientIp,State,ShellRunTime,ShellInactivity

ShellId                              ClientIP      State        ShellRunTime ShellInactivity
-------                              --------      -----        ------------ ---------------
BE79B3CD-A866-4593-B140-21A8E154DDE8 192.168.181.1 Connected    P0DT0H9M27S  P0DT0H0M6S

Finally, here is my traceback:

Traceback

  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_exceptions.py", line 10, in map_exceptions
    yield
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_backends\sync.py", line 126, in read
    return self._sock.recv(max_bytes)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "F:\Documents\GitHub\Devious-WinRM\src\devious_winrm\test.py", line 26, in <module>
    output = ps.add_script("date").add_command("Out-String").invoke()
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 1137, in invoke
    output_task = self.invoke_async(
        input_data=input_data,
        output_stream=output_stream,
        buffer_input=buffer_input,
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 1634, in invoke_async
    return super().invoke_async(input_data, output_stream, completed, buffer_input)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 1204, in invoke_async
    connection.command(self._pipeline.pipeline_id)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_connection\wsman.py", line 524, in command
    resp = self._connection.post(self._shell.data_to_send())
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 1064, in post
    response = self._http.post(
        self.connection_uri,
    ...<4 lines>...
        extensions=ext,
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1132, in post
    return self.request(
           ~~~~~~~~~~~~^
        "POST",
        ^^^^^^^
    ...<11 lines>...
        extensions=extensions,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 901, in send
    response = self._send_handling_auth(
        request,
    ...<2 lines>...
        history=[],
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
        request,
        follow_redirects=follow_redirects,
        history=history,
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 679, in handle_request
    resp = self._connection.handle_request(req)
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\connection.py", line 103, in handle_request
    return self._connection.handle_request(request)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 133, in handle_request
    raise exc
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 111, in handle_request
    ) = self._receive_response_headers(**kwargs)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 176, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 212, in _receive_event
    data = self._network_stream.read(
        self.READ_NUM_BYTES, timeout=timeout
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_backends\sync.py", line 124, in read
    with map_exceptions(exc_map):
         ~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\Pablo\AppData\Local\Programs\Python\Python313\Lib\contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\Documents\GitHub\Devious-WinRM\src\devious_winrm\test.py", line 17, in <module>
    with SyncRunspacePool(conn) as rp:
         ~~~~~~~~~~~~~~~~^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 519, in __exit__
    self.close()
    ~~~~~~~~~~^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 689, in close
    connection.close()
    ~~~~~~~~~~~~~~~~^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_connection\wsman.py", line 497, in close
    resp = self._connection.post(self._shell.data_to_send())
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 1064, in post
    response = self._http.post(
        self.connection_uri,
    ...<4 lines>...
        extensions=ext,
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1132, in post
    return self.request(
           ~~~~~~~~~~~~^
        "POST",
        ^^^^^^^
    ...<11 lines>...
        extensions=extensions,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 814, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 901, in send
    response = self._send_handling_auth(
        request,
    ...<2 lines>...
        history=[],
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
        request,
        follow_redirects=follow_redirects,
        history=history,
    )
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 679, in handle_request
    resp = self._connection.handle_request(req)
  File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\connection.py", line 101, in handle_request
    raise ConnectionNotAvailable()
httpcore.ConnectionNotAvailable

[1upbyte]

Any updates @jborean93? If you have an idea as to what might be causing the issue let me know, I have some free time that I could use to squash this bug.

[adityatelange]

@1upbyte see #21 (comment)

[1upbyte]

This is not the behavior I'm experiencing on the beta release. In my case the RunspacePool doesn't become disconnected.

I did manage a workaround by having a thread in the background create a new Pipeline using the RunspacePool and sending an empty script every 60 seconds.

[jborean93]

I'm not ignoring this sorry it is just a bit low in my priority list right now. The whole connection handling probably needs an overhaul and while I've got a POC branch that drops httpx in favour of using httpcore directly I've not had the time to properly have it deal with things like timeouts and the like.