Skip to content

Feature/demand assessor#49

Open
AneeshD04 wants to merge 4 commits into
aims-foundations:mainfrom
AneeshD04:feature/demand-assessor
Open

Feature/demand assessor#49
AneeshD04 wants to merge 4 commits into
aims-foundations:mainfrom
AneeshD04:feature/demand-assessor

Conversation

@AneeshD04

Copy link
Copy Markdown
  • DemandAssessor: neural IRT model predicting P(response=1 | subject, item_features) via MLP over concatenated subject embeddings and item feature vectors
  • Full demand annotation pipeline (DemandAnnotator, GeminiClient, RubricsCatalog, AnnotationCache) implementing the 18-dimension ADeLe rubric scoring system (Annotator modeled after mafter Nature paper “General scales unlock AI evaluation with explanatory and predictive power")
  • Unit tests for DemandAssessor (24 tests, synthetic data, no pretrained model needed)
  • Unit and live end-to-end tests for annotation pipeline
  • Fix deferred import in LLMJudge to avoid crash when transformers not installed

For Issue #41

- DemandAssessor: neural IRT model predicting P(response=1 | subject, item_features)
  via MLP over concatenated subject embeddings and item feature vectors
- Full demand annotation pipeline (DemandAnnotator, GeminiClient, RubricsCatalog,
  AnnotationCache) implementing the 18-dimension ADeLe rubric scoring system
- Unit tests for DemandAssessor (24 tests, synthetic data, no pretrained model needed)
- Unit and live end-to-end tests for annotation pipeline
- Fix deferred import in LLMJudge to avoid crash when transformers not installed

Closes aims-foundations#41
@AneeshD04 AneeshD04 force-pushed the feature/demand-assessor branch from 6a61573 to e011be2 Compare June 16, 2026 04:47
…y client agnostic (use environment vars to set client and key
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant