SageMaker AI Fine-tuning: Misleading evaluation dataset format documentation

[rpradyrao]

Product Version

• Amazon SageMaker Studio Classic
• Amazon SageMaker Studio
• Issue is not related to SageMaker Studio

Issue Description

Problem

Unable to upload evaluation datasets for fine-tuning in SageMaker AI. All formats fail validation with error: "Invalid dataset. Does not match any expected format."

What We Tried

Format 1: UI Examples (with system prompt)
The UI shows this format:

{
  "system": "You are an English major with top marks in class who likes to give minimal word responses: ",
  "query": "What is the symbol that ends the sentence as a question",
  "response": "?"
}

Result: "Invalid dataset. Does not match any expected format"

Format 2: UI Examples (single line JSONL)

{"system": "You are an English major...", "query": "What is the symbol...", "response": "?"}

Result: "Invalid dataset. Does not match any expected format"

Format 3: Documentation Format (prompt/referenceResponse)
Based on https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-prompt-datasets-custom.html:

{"prompt": "What is the symbol that ends the sentence as a question", "referenceResponse": "?"}

Result: "Invalid dataset. Does not match any expected format"

Format 4: Pretty-printed JSON
Multi-line formatted JSON objects with blank lines between
Result: "Invalid JSON format in file"

Format 5: JSONL without blank lines
Consecutive JSON objects, one per line
Result: "Invalid dataset. Does not match any expected format"

Steps to Reproduce

1. Go to SageMaker AI fine-tuning interface
2. Navigate to evaluation dataset upload
3. Create a .jsonl file with ANY of the formats above (including exact UI examples)
4. Attempt to upload
5. Receive error: "Invalid dataset. Does not match any expected format"

Request

Please provide:

1. A working example file that passes validation
2. Complete format specification with all required/optional fields
3. Fix for the upload validator or update documentation to match actual requirements

Expected Behavior

Expected Behavior

• At minimum, the exact examples shown in the UI should work
• OR the documentation format should work
• OR there should be clear error messages explaining what's wrong

Observed Behavior

Actual Behavior

• Every format fails validation
• Even copying the exact examples from the UI fails
• No specific error message about what's wrong with the format
• Generic error provides no actionable guidance

Impact

• Blocker: Cannot upload evaluation datasets at all
• Affects both custom evaluation datasets and attempts to use documented formats
• No workaround available
• Makes fine-tuning evaluation not possible

Product Category

Models

Feedback Category

Configuration and Setup

Other Details

No response