Features Backlog

[luca-martial]

Parked Ideas 🚗

• Testing social stereotypes for MLMs (paper)
• Adding support for cloud provider models to be tested
• Support an installation for airgapped environments
• Need to generate a spec of what data the model expects and can safely run with (for example, if a model has only been validated on females aged 18 and up, then the model should not be used on people outside that demographic group) - see https://ianwhitestone.work/hello-great-expectations/
• Toxicity tests (swear words, offensive answers)
• Data leakage tests (PHI)
• Adversarial attacks tests
• Freshness tests (replace _2023_name)
• Runtime tests
• Question answering
• Text generation
• Summarization
• Paraphrasing
• Translation