[BUG] [v0.0.7] JSON agent prompt.md fails on UTF-16 encoding

[EnthusiasticTech]

Project

cortex

Description

JSON agents support UTF‑16/UTF‑8‑BOM via read_file_with_encoding, but their prompt.md is read with std::fs::read_to_string, which only accepts UTF‑8. A UTF‑16 prompt file causes the entire agent load to fail, even though UTF‑16 is supported for the agent definition itself.

Reference: cortex-cli/src/agent_cmd/loader.rs load_agent_from_json uses read_to_string for prompt.md.

Error Message

failed to read .../prompt.md: stream did not contain valid UTF-8

Debug Logs

N/A

System Information

Bounty Version: 0.1.0
OS: Ubuntu 24.04 LTS
CPU: AMD EPYC-Genoa Processor (8 cores)
RAM: 15 GB

Screenshots

No response

Steps to Reproduce

1. Create a directory agent at ~/.cortex/agents/utfprompt/agent.json:

   {
     "name": "utfprompt",
     "description": "UTF prompt agent",
     "mode": "primary"
   }

2. Create ~/.cortex/agents/utfprompt/prompt.md encoded as UTF‑16 LE with BOM (e.g., via iconv -f utf-8 -t utf-16le or a Windows editor) and add any text.
3. Run: cortex agent list
4. Observe that the agent fails to load with a UTF‑8 decoding error.

Expected Behavior

prompt.md should be read using the same encoding-aware logic as agent definitions (UTF‑8/UTF‑16/BOM), or at least not prevent agent loading.

Actual Behavior

UTF‑16 prompt files cause agent loading to fail because read_to_string only accepts UTF‑8.

Additional Context

This is inconsistent with read_file_with_encoding already used for agent.json and AGENT.md.

[hechibing]

I’d like to work on this bounty.

[hechibing]

Implemented fix in Cortex repo and opened PR: CortexLM/cortex#152

What changed:

• switched JSON agent prompt.md loading to read_file_with_encoding (UTF-8/UTF-16/BOM aware)
• aligned prompt loading behavior with existing encoding-aware JSON/MD loader paths
• added regression test for UTF-16 LE + BOM prompt.md loading

Validation run:

• cargo test -p cortex-cli test_load_agent_from_json_with_utf16_prompt_file -- --nocapture