cert: runners for metric 5 (model support) and metric 6 (dataset support) - #2
Open
kimminhyun-ai wants to merge 3 commits into
Open
kimminhyun-ai wants to merge 3 commits into
kimminhyun-ai wants to merge 3 commits into
Conversation
…ort) Metric 5 counts unique Hugging Face model IDs that actually answer. The runtime here is HuggingFace transformers behind an OpenAI-compatible API, holding one model at a time and reporting the repository ID it really loaded. A request naming a model it is not holding is refused rather than answered by whatever is loaded — without that, the count is "how many replies came back", not "how many models are supported". Base models that were never instruction-tuned emit no end token and carry on writing the other side of the conversation. The server reads the turn boundary out of each model's own chat template and stops there; a model with no template gets a completion framing and stops at the line end. That changes how the question is put to the model, not which model answers it. Metric 6 counts datasets that went all the way: import verified, training READY, knowledge published, and every check question answered with the knowledge applied. The planner reads each repository's real viewer to pick the config, split and columns, and drops two kinds of source before any time is spent on them — datasets whose only answer column is a label (training on those teaches the model to answer "1") and datasets that repeat a question with different answers (the node rejects the whole file as dataset_empty). Both runners record the command, the identifiers and the failure reason per item, and count the stages separately so an import cannot pass for an integration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019hZqdLKnVfzvRf9mus6NNh
Three things were being learned the slow way, one dataset or model at a time,
after the import or the inference had already run.
Column values that are not scalars. `answers: {"text": [...], "answer_start":
[...]}` reads fine in the viewer and is refused at import with "must be a
scalar, not an object or array". The planner now requires the chosen columns to
be scalar in every sampled row.
The training queue is one job wide. `--train --wait` gave up after its timeout
while the job sat QUEUED behind another node's lesson, and the result was lost
even though the job itself carried on. Submission and observation are now
separate: submit, then watch that job id to a terminal state.
The submitted job id. `--train` returns the job nested inside `job`, so reading
only the outer object found no id and the whole training step was silently
skipped.
Also: a model repository over a size limit is skipped rather than downloaded,
and completion-style models stop at the next question rather than writing the
rest of the interview themselves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019hZqdLKnVfzvRf9mus6NNh
The CLI can emit a truncated multibyte sequence, and `subprocess.run(text=True)` raises UnicodeDecodeError on it — killing the whole driver partway through a 150-item sweep over one item's output. Decoding now replaces the bad bytes, so the item is recorded as it came back and the run continues. Also adds cert/m4: the five Ainize nodes the inference load test needs. They read the AIN chain (inference records refuse a local ledger), point at a real serving runtime, and have teaching disabled — a lesson would take the runtime's exclusive lock and stop the very thing being measured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019hZqdLKnVfzvRf9mus6NNh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two runners for the 3rd-year certification metrics that had none, each counting what the metric actually asks for rather than what is easy to count.
Metric 5 — model support. A HuggingFace transformers runtime behind an OpenAI-compatible API holds one model at a time and reports the repository ID it really loaded. A request naming a model it is not holding is refused, not answered by whatever happens to be loaded. Without that the count is "how many replies came back", not "how many models are supported". Base models that were never instruction-tuned emit no end token and write the other side of the conversation; the server reads the turn boundary out of each model's own chat template and stops there.
Metric 6 — dataset support. Counts only datasets that went all the way: import verified, training
READY, knowledge published, every check question answered with the knowledge applied. The planner reads each repository's real dataset viewer to choose config, split and columns, and drops two kinds of source before any time is spent on them — datasets whose only answer column is a label, and datasets that repeat a question with different answers.Both record the command, the identifiers and the failure reason per item, and count the stages separately so an import cannot pass for an integration.
Metric 5 needs ainize-node#12: before that fix a node refuses base-model chat on any runtime without the patch hook.
🤖 Generated with Claude Code
https://claude.ai/code/session_019hZqdLKnVfzvRf9mus6NNh