UTF characters in input cause parsing error in an environment without locales configured

[head-gardener]

Flake
I originally encountered this error when using neorg-pandoc in a Nix build sandbox, which doesn't configure locales by default. Below is a flake that can be used to reproduce the issue.

{
  inputs = {
    flake-parts.url = "github:hercules-ci/flake-parts";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      systems =
        [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin" ];
      perSystem = { pkgs, lib, ... }: {
        packages = rec {
          neorg-haskell-parser = let
            base = pkgs.haskellPackages.callPackage ({ mkDerivation, aeson, base
              , bytestring, containers, directory, fetchgit, filepath, hspec
              , HUnit, lib, megaparsec, optics-core, optparse-applicative
              , pandoc-types, text, transformers, typed-process }:
              mkDerivation {
                pname = "neorg";
                version = "0.2.0.0";
                src = fetchgit {
                  url = "https://github.com/Simre1/neorg-haskell-parser";
                  sha256 =
                    "0gvwkdd3ga9251z8n6zk0wjcdxyga2nlg38r8ifhyzl14hl752il";
                  rev = "a96dc88354492c8112b0c7d7a0370e07dbf49b6d";
                  fetchSubmodules = true;
                };
                isLibrary = true;
                isExecutable = true;
                libraryHaskellDepends = [
                  base
                  containers
                  megaparsec
                  optics-core
                  pandoc-types
                  text
                  transformers
                ];
                executableHaskellDepends = [
                  aeson
                  base
                  bytestring
                  containers
                  optics-core
                  optparse-applicative
                  pandoc-types
                  text
                  transformers
                ];
                testHaskellDepends = [
                  aeson
                  base
                  containers
                  directory
                  filepath
                  hspec
                  HUnit
                  optics-core
                  text
                  transformers
                  typed-process
                ];
                homepage = "https://github.com/Simre1/neorg-haskell-parser";
                description = "Transform norg documents into PDFs with pandoc";
                license = lib.licenses.mit;
                mainProgram = "neorg-pandoc";
              }) { };
          in lib.pipe base (with pkgs.haskell.lib; [
            doJailbreak
            dontCheck
            dontHaddock
            justStaticExecutables
          ]);

          default = pkgs.stdenvNoCC.mkDerivation {
            pname = "test";
            version = "20250105";

            src = ./.;

            buildInputs = [ neorg-haskell-parser ];

            installPhase = ''
              echo '⭐️' | neorg-pandoc -i > $out
            '';
          };
        };
      };
      flake = { };
    };
}

Complete flake provided for transparency. Important part is

echo '⭐️' | neorg-pandoc -i > $out

Problem & Expected behavior
In a normal environment this produces {"pandoc-api-version":[1,23,1],"meta":{},"blocks":[{"t":"Para","c":[{"t":"Str","c":"⭐️"}]}]} . Under Nix build sandbox, however, this fails with:

neorg-pandoc: <stdin>: hGetContents: invalid argument (cannot decode byte sequence starting from 226)

Additional context
The issue is with text's readFile and getContents, which don't like UTF-8 in absence of properly configured locales.

Working fixes:

• Using Data.Text.IO.Utf8 from text-2.1.
• Setting locales in the build environment.

Still, the error is pretty confusing, especially if you don't know what characters are causing it.