Decode command output as UTF-8; stop guessing the remote code page (#142)

README.md

@@ -69,9 +69,39 @@ Connection-scoped options on the builder: `https()`, `port(int)`,
 
 Per-operation options: `namespace(...)`, `timeout(...)`, and for WQL enumeration tuning
 `pageSize(int)` (WS-Enumeration `MaxElements`, 32000 by default) and `pullTimeout(Duration)`
-(`MaxTime` per Pull). Commands accept `workingDirectory(String)`, `charset(Charset)` (detected
-from the remote code set by default), and `upload(Path...)` to copy local script files and rewrite
-the command to reference the remote copies.
+(`MaxTime` per Pull). Commands accept `workingDirectory(String)`, `charset(Charset)` (see
+[Character encoding](#character-encoding)), and `upload(Path...)` to copy local script files and
+rewrite the command to reference the remote copies.
+
+### Character encoding
+
+The remote command shell is created with console code page **65001**, so command output is decoded
+as **UTF-8** — whatever the remote machine's locale. Accented and non-Latin characters come back
+exactly as the remote host wrote them, and nothing needs to be detected or configured:
+
+```java
+// On a French Windows host:
+client.command("vol").execute().stdout();   // " Le numéro de série du volume est …"
+```
+
+A few legacy console tools — `net.exe` and `chcp.com` are the known ones — ignore the console code
+page and write text pre-converted to the machine's **OEM** code page. Their accented characters
+cannot be decoded as UTF-8 (they arrive as `U+FFFD`); decode such a command with the matching OEM
+charset instead:
+
+```java
+client.command("net user Administrateur")
+    .charset(Charset.forName("IBM850"))     // French/Western European OEM code page
+    .execute();
+```
+
+WQL results are unaffected: they travel as UTF-8 inside the SOAP envelope.
+
+> **Changed in 2.0.00** — earlier versions ran a `SELECT CodeSet FROM Win32_OperatingSystem` query
+> before each command and decoded the output with the code page it reported. That is the remote
+> machine's *ANSI* code page, which never matched what the shell emitted, so all non-ASCII output
+> was mangled on non-English hosts. `WindowsRemoteProcessUtils.getWindowsEncodingCharset()` is
+> deprecated as a result, and the extra query is gone.
 
 ### Streaming
 

src/main/java/org/metricshub/winrm/CommandRequest.java

@@ -33,7 +33,6 @@
 import org.metricshub.winrm.exceptions.WinRMClientException;
 import org.metricshub.winrm.exceptions.WinRMTimeoutException;
 import org.metricshub.winrm.exceptions.WindowsRemoteException;
-import org.metricshub.winrm.exceptions.WqlQuerySyntaxException;
 
 /**
  * A command being prepared for execution, created by {@link WinRMClient#command(String)}.
@@ -81,7 +80,7 @@ public CommandRequest workingDirectory(final String workingDirectory) {
 
 	/**
 	 * Set the timeout of this command. For {@link #execute()} it is a wall-clock deadline covering
-	 * file uploads, encoding detection, and the command itself; for {@link #start()} it is an
+	 * file uploads and the command itself; for {@link #start()} it is an
 	 * <i>inactivity</i> timeout — the longest silence tolerated from the server between two
 	 * responses, with no overall deadline. Default: the client's timeout.
 	 *
@@ -94,8 +93,11 @@ public CommandRequest timeout(final Duration timeout) {
 	}
 
 	/**
-	 * Set the charset used to decode the command output. Default: detected from the remote
-	 * operating system's code set (one extra WQL query, cached on the client).
+	 * Set the charset used to decode the command output. Default:
+	 * {@link WindowsRemoteExecutor#SHELL_OUTPUT_CHARSET} (UTF-8), which is what the remote shell
+	 * emits — its console code page is set to 65001 when the shell is created, whatever the remote
+	 * locale. Override this only for a command that changes the console code page itself (a leading
+	 * {@code chcp}) or writes raw bytes in another encoding to its standard output.
 	 *
 	 * @param charset the output charset
 	 * @return this request
@@ -212,7 +214,7 @@ public CommandResult execute() {
 			return Utils.execute(() -> drainWithCallbacks(prepared, remaining, start), remaining);
 		} catch (final TimeoutException e) {
 			throw timeoutException(e);
-		} catch (final IOException | WqlQuerySyntaxException e) {
+		} catch (final IOException e) {
 			throw new WinRMClientException(e.getMessage(), e);
 		} catch (final InterruptedException e) {
 			Thread.currentThread().interrupt();
@@ -241,8 +243,8 @@ public CommandResult execute() {
 	 * <b>The process must be closed</b> — use try-with-resources. It holds the client's serial
 	 * connection until the command completes or the handle is closed; closing early terminates
 	 * the remote command (WinRM terminate {@code Signal}). The timeout acts as an
-	 * <i>inactivity</i> timeout — see {@link RemoteProcess}. File uploads and encoding detection
-	 * run here, before the command starts.
+	 * <i>inactivity</i> timeout — see {@link RemoteProcess}. File uploads run here, before the
+	 * command starts.
 	 *
 	 * @return the running process handle, to use with try-with-resources
 	 * @throws org.metricshub.winrm.exceptions.WinRMTimeoutException when the command startup times out
@@ -263,7 +265,7 @@ public RemoteProcess start() {
 			return new RemoteProcess(cursor, prepared.charset, client.hostname(), timeout);
 		} catch (final TimeoutException e) {
 			throw timeoutException(e);
-		} catch (final IOException | WqlQuerySyntaxException e) {
+		} catch (final IOException e) {
 			throw new WinRMClientException(e.getMessage(), e);
 		} catch (final WindowsRemoteException e) {
 			throw WinRMClient.translate(e);
@@ -290,7 +292,7 @@ private static final class Prepared {
 	 * resolve the output charset.
 	 */
 	private Prepared prepare(final long timeoutMillis, final long start)
-		throws IOException, TimeoutException, WqlQuerySyntaxException, WindowsRemoteException {
+		throws IOException, TimeoutException, WindowsRemoteException {
 		String actualCommand = commandLine;
 		String actualWorkingDirectory = workingDirectory;
 
@@ -309,7 +311,7 @@ private Prepared prepare(final long timeoutMillis, final long start)
 			actualWorkingDirectory = null;
 		}
 
-		final Charset actualCharset = charset != null ? charset : client.detectCharset(timeoutMillis, start);
+		final Charset actualCharset = charset != null ? charset : WindowsRemoteExecutor.SHELL_OUTPUT_CHARSET;
 		return new Prepared(actualCommand, actualWorkingDirectory, actualCharset);
 	}
 

src/main/java/org/metricshub/winrm/WinRMClient.java

@@ -22,7 +22,6 @@
 
 import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 import java.io.IOException;
-import java.nio.charset.Charset;
 import java.nio.file.Path;
 import java.time.Duration;
 import java.util.ArrayList;
@@ -33,7 +32,6 @@
 import org.metricshub.winrm.exceptions.WinRMException;
 import org.metricshub.winrm.exceptions.WinRMTimeoutException;
 import org.metricshub.winrm.exceptions.WindowsRemoteException;
-import org.metricshub.winrm.exceptions.WqlQuerySyntaxException;
 import org.metricshub.winrm.service.WinRMEndpoint;
 import org.metricshub.winrm.service.WinRMExecutorFactory;
 import org.metricshub.winrm.service.client.auth.AuthenticationEnum;
@@ -85,10 +83,6 @@ public final class WinRMClient implements AutoCloseable {
 	private final String namespace;
 	private final Duration timeout;
 
-	// The remote OS code set does not change during a session: detect it once, on the first
-	// command that needs it, and reuse it for every later command.
-	private volatile Charset detectedCharset;
-
 	private WinRMClient(
 		final WindowsRemoteExecutor executor,
 		final String hostname,
@@ -192,23 +186,6 @@ Duration defaultTimeout() {
 		return timeout;
 	}
 
-	/**
-	 * The charset of the remote command output, detected from the remote OS code set on first
-	 * use and cached for the lifetime of the client.
-	 */
-	Charset detectCharset(final long timeoutMillis, final long start)
-		throws TimeoutException, WqlQuerySyntaxException, WindowsRemoteException {
-		Charset charset = detectedCharset;
-		if (charset == null) {
-			charset = WindowsRemoteProcessUtils.getWindowsEncodingCharset(
-				executor,
-				TimeoutHelper.getRemainingTime(timeoutMillis, start, "No time left to detect the remote encoding")
-			);
-			detectedCharset = charset;
-		}
-		return charset;
-	}
-
 	/**
 	 * Translate a legacy checked exception into the unchecked hierarchy: when a typed
 	 * {@link WinRMClientException} (fault, authentication) is in the cause chain, it is unwrapped

src/main/java/org/metricshub/winrm/WindowsRemoteExecutor.java

@@ -21,6 +21,7 @@
  */
 
 import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.TimeoutException;
@@ -34,6 +35,13 @@ public interface WindowsRemoteExecutor extends AutoCloseable {
 	 */
 	int DEFAULT_WQL_MAX_ELEMENTS = 32000;
 
+	/**
+	 * Charset of the output of every command run through this executor: UTF-8, because the remote
+	 * command shell is created with console code page 65001. It is not the remote machine's ANSI or
+	 * OEM code page, and it does not depend on the remote locale.
+	 */
+	Charset SHELL_OUTPUT_CHARSET = StandardCharsets.UTF_8;
+
 	/**
 	 * <p>
 	 * Execute a WQL query and process its result.
@@ -151,7 +159,8 @@ default CommandCursor startCommand(final String command, final String workingDir
 	 *
 	 * @param command The command to execute
 	 * @param workingDirectory Path of the directory for the spawned process on the remote system (can be null)
-	 * @param charset The charset
+	 * @param charset The charset decoding the command output; {@code null} uses
+	 *        {@link #SHELL_OUTPUT_CHARSET}, which is what the remote shell actually emits
 	 * @param timeout Timeout in milliseconds
 	 * @return The command result
 	 * @throws WindowsRemoteException For any problem encountered

src/main/java/org/metricshub/winrm/WindowsRemoteProcessUtils.java

@@ -86,7 +86,15 @@ private WindowsRemoteProcessUtils() {}
 	 * @throws WindowsRemoteException For any problem encountered on remote
 	 * @see <a href="https://docs.microsoft.com/en-us/windows/win32/cimwin32prov/win32-operatingsystem">
 	 *      Win32_OperatingSystem class</a>
+	 * @deprecated Not the charset of remote command output, and never was: {@code CodeSet} is the
+	 *             remote machine's <em>ANSI</em> code page, while {@code cmd.exe} writes its output in
+	 *             the <em>console (OEM)</em> code page — the two differ on every non-English locale.
+	 *             Command output is now decoded with
+	 *             {@link WindowsRemoteExecutor#SHELL_OUTPUT_CHARSET}, the code page the remote shell
+	 *             is created with. Use this method only to decode data that a Windows application
+	 *             genuinely wrote in the ANSI code page.
 	 */
+	@Deprecated(since = "2.0.00", forRemoval = false)
 	public static Charset getWindowsEncodingCharset(
 		final WindowsRemoteExecutor windowsRemoteExecutor,
 		final long timeout

src/main/java/org/metricshub/winrm/command/WinRMCommandExecutor.java

@@ -21,7 +21,6 @@
  */
 
 import java.io.IOException;
-import java.nio.charset.Charset;
 import java.nio.file.Path;
 import java.util.Collections;
 import java.util.List;
@@ -33,9 +32,7 @@
 import org.metricshub.winrm.WinRMHttpProtocolEnum;
 import org.metricshub.winrm.WindowsRemoteCommandResult;
 import org.metricshub.winrm.WindowsRemoteExecutor;
-import org.metricshub.winrm.WindowsRemoteProcessUtils;
 import org.metricshub.winrm.exceptions.WindowsRemoteException;
-import org.metricshub.winrm.exceptions.WqlQuerySyntaxException;
 import org.metricshub.winrm.service.WinRMEndpoint;
 import org.metricshub.winrm.service.WinRMExecutorFactory;
 import org.metricshub.winrm.service.client.auth.AuthenticationEnum;
@@ -108,12 +105,12 @@ public static WindowsRemoteCommandResult execute(
 				authentications
 			)) {
 			if (localFiles.isEmpty()) {
-				final Charset charset = WindowsRemoteProcessUtils.getWindowsEncodingCharset(
-					winRMService,
-					TimeoutHelper.getRemainingTime(timeout, start, "No time left to retrieve the code set")
+				return winRMService.executeCommand(
+					command,
+					workingDirectory,
+					WindowsRemoteExecutor.SHELL_OUTPUT_CHARSET,
+					timeout
 				);
-
-				return winRMService.executeCommand(command, workingDirectory, charset, timeout);
 			}
 
 			// Copy the specified list of files through the command shell, and update the command accordingly
@@ -124,19 +121,12 @@ public static WindowsRemoteCommandResult execute(
 				TimeoutHelper.getRemainingTime(timeout, start, "No time left to copy the local files")
 			);
 
-			final Charset charset = WindowsRemoteProcessUtils.getWindowsEncodingCharset(
-				winRMService,
-				TimeoutHelper.getRemainingTime(timeout, start, "No time left to retrieve the code set")
-			);
-
 			return winRMService.executeCommand(
 				String.format("CMD.EXE /C (%s)", localFilesUpdatedCommand),
 				null,
-				charset,
+				WindowsRemoteExecutor.SHELL_OUTPUT_CHARSET,
 				TimeoutHelper.getRemainingTime(timeout, start, "No time left to execute command")
 			);
-		} catch (final WqlQuerySyntaxException e) {
-			throw new IOException(e);
 		}
 	}
 }

src/main/java/org/metricshub/winrm/light/Envelopes.java

@@ -55,6 +55,13 @@ final class Envelopes {
 
 	private static final int MAX_ENVELOPE_SIZE = 153600;
 
+	/**
+	 * Console code page of the remote shell: UTF-8. The shell's output charset must be one the
+	 * client knows without asking, and it must be able to carry every locale's characters — an OEM
+	 * page like 437 can encode neither {@code é} on a French host nor anything CJK at all.
+	 */
+	private static final String CODEPAGE_UTF8 = "65001";
+
 	private Envelopes() {}
 
 	// --- WQL ---------------------------------------------------------------
@@ -121,7 +128,7 @@ static String release(final String url, final String namespace, final String con
 	static String createShell(final String url, final String workingDirectory, final long timeoutMs) {
 		final String optionSet = "<wsman:OptionSet>" +
 			"<wsman:Option Name=\"WINRS_NOPROFILE\">TRUE</wsman:Option>" +
-			"<wsman:Option Name=\"WINRS_CODEPAGE\">437</wsman:Option>" +
+			"<wsman:Option Name=\"WINRS_CODEPAGE\">" + CODEPAGE_UTF8 + "</wsman:Option>" +
 			"</wsman:OptionSet>";
 		final String workingDir = (workingDirectory == null || workingDirectory.trim().isEmpty())
 			? ""

src/main/java/org/metricshub/winrm/light/WsmanClient.java

@@ -36,6 +36,7 @@
 import javax.xml.XMLConstants;
 import javax.xml.parsers.DocumentBuilder;
 import javax.xml.parsers.DocumentBuilderFactory;
+import org.metricshub.winrm.WindowsRemoteExecutor;
 import org.metricshub.winrm.exceptions.WinRMAuthenticationException;
 import org.metricshub.winrm.exceptions.WinRMFaultException;
 import org.w3c.dom.Document;
@@ -413,7 +414,8 @@ static final class CommandOutput {
 	 *
 	 * @param commandLine the command line to run
 	 * @param workingDirectory working directory of the shell (only honored when the shell is created)
-	 * @param charset the charset decoding the output streams
+	 * @param charset the charset decoding the output streams; {@code null} uses
+	 *        {@link WindowsRemoteExecutor#SHELL_OUTPUT_CHARSET}
 	 * @param operationTimeoutMs this operation's timeout, driving the WSMan OperationTimeout header
 	 *        and the socket read timeout
 	 */
@@ -423,7 +425,7 @@ CommandOutput executeCommand(
 		final Charset charset,
 		final long operationTimeoutMs
 	) throws Exception {
-		final Charset cs = charset != null ? charset : StandardCharsets.UTF_8;
+		final Charset cs = charset != null ? charset : WindowsRemoteExecutor.SHELL_OUTPUT_CHARSET;
 		final ByteArrayOutputStream stdout = new ByteArrayOutputStream();
 		final ByteArrayOutputStream stderr = new ByteArrayOutputStream();
 		try (RemoteCommand command = startCommand(commandLine, workingDirectory, operationTimeoutMs, false)) {

src/site/markdown/cli.md

@@ -88,8 +88,10 @@ failure can therefore leave partial output on standard output, signalled by the
 
 Remote stdout and stderr are forwarded **live** to the corresponding local streams while the
 command runs — each chunk is flushed as it arrives, so a long-running command can be followed in
-real time. The output is decoded with the remote host's active code page, detected automatically
-before the command starts.
+real time. The output is decoded as UTF-8: the remote shell is created with console code page
+65001, so no code-page detection is needed and non-ASCII output survives whatever the remote
+locale. See [Character encoding](commands.html#character-encoding) for the two legacy tools that
+ignore the console code page.
 
 ## Timeout semantics
 
@@ -98,8 +100,8 @@ before the command starts.
 * For `wql`, it is the **inactivity timeout** of the stream — the longest tolerated silence
   between two server responses. A large result can stream for longer than the timeout, as long as
   the server keeps answering.
-* For `command`, it is the **overall deadline** covering the encoding detection and the command
-  itself.
+* For `command`, it is the **overall deadline** covering the command itself and any file
+  uploads.
 
 See [Timeouts and Errors](timeouts-and-errors.html) for the underlying semantics.