Malania 2007 benchmark#365
Merged
Merged
Conversation
…nchmark, packaging and a few tests.
…alsError on trying to access them
…emoval of pool_score
Contributor
|
Hi @benlonnqvist, we're doing some spring cleaning on the PR backlog and noticed that this PR is active and passing tests! Is it ready to be reviewed and merged? Thanks for the contributions! |
Contributor
Author
|
Hi @deirdre-k, thanks for messaging! Let me update the branch to double check that #917 didn't cause any issues, add one more test, and if it passes after that, it should be good to go! Sorry about not having it tagged as a draft, I'll ping you later today/this week when it's good to go. |
Contributor
|
No worries at all, sounds great! And thanks for the quick reply 😀 |
mschrimpf
approved these changes
Jul 5, 2024
mschrimpf
left a comment
mschrimpf
left a comment
Member
There was a problem hiding this comment.
Required change: remove aggregation dimension from all scores.
Recommended change: update naming convention to reserve - for benchmark-metric separation only (use . for sub-data instead)
| return ceiling | ||
|
|
||
| @staticmethod | ||
| def compute_threshold_elevations(assemblies: Dict[str, PropertyAssembly]) -> list: |
Member
There was a problem hiding this comment.
Suggested change
| def compute_threshold_elevations(assemblies: Dict[str, PropertyAssembly]) -> list: | |
| def compute_threshold_elevations(assemblies: Dict[str, PropertyAssembly]) -> List: |
| # independent_variable is not used since we compute from thresholds, and do not need to fit them | ||
| metric = load_metric('threshold', independent_variable='placeholder') | ||
| score = metric(float(assembly.sel(subject='A').values), assembly) | ||
| print(score) |
| baseline_condition='placeholder', | ||
| test_condition='placeholder') | ||
| score = metric(float(assembly.sel(subject='A').values), assembly) | ||
| print(score) |
Comment on lines
+188
to
+190
| score = float(score[(score['aggregation'] == 'center')].values) | ||
| human_thresholds.append(random_human_score) | ||
| scores.append(score) |
Member
There was a problem hiding this comment.
Suggested change
| score = float(score[(score['aggregation'] == 'center')].values) | |
| human_thresholds.append(random_human_score) | |
| scores.append(score) | |
| human_thresholds.append(random_human_score) | |
| scores.append(score.values) |
Comment on lines
+492
to
+494
| score = float(score[(score['aggregation'] == 'center')].values) | ||
| human_threshold_elevations.append(random_human_score) | ||
| scores.append(score) |
Member
There was a problem hiding this comment.
Suggested change
| score = float(score[(score['aggregation'] == 'center')].values) | |
| human_threshold_elevations.append(random_human_score) | |
| scores.append(score) | |
| human_threshold_elevations.append(random_human_score) | |
| scores.append(score.values) |
Co-authored-by: Martin Schrimpf <mschrimpf@users.noreply.github.com>
Contributor
Author
Thanks @mschrimpf for the review. I implemented both sets of changes and pending jenkins plugin tests, all good to go from my side. |
Nishant-codex
pushed a commit
to Nishant-codex/vision
that referenced
this pull request
Jul 2, 2026
* Initial Malania 2007 commit with preliminary threshold metric, the benchmark, packaging and a few tests. * Add correct visual angle * clean up threshold metric file * Further threshold cleanup and ceiling compute verification * Remove python 3.8 requirement that came from typing.Literal * clean up benchmark file * add assembly, stimulus, and benchmark general tests * resolve Islam2021 conflict * Add grid search, correct psychometric function, and pooled score * Standardized vernier train and test set sizes * add goodness of fit check to discard fits to random responses * update ceilings to pooled expectations * packaged commit * add Malania2007 entries to lookup.csv * add Malania2007 to benchmark init * fix indexing error in test___init__ * add missing year(s) in identifiers where they were missing * add S3-related tests to private access, as Travis throws a NoCredentialsError on trying to access them * minor cleanup of threshold * add a few comments to threshold functions * fix typo in precomputed feature test * fix typos in dataset and assembly call functions * Update Assembly type and name * update scoring method with comment and update ceilings * update benchmark to 2.0 format w/ local test * update data registry format * remove BIBTEX from threshold __init__ * update metric imports to 2.0 * re-add test_stimuli.py deleted by accident * change nan filtering to subject_unique_id filtering * add require_variance to model call * update stimulus and data assembly related information with arcane knowledge collected from years of hard work * more arcane knowledge, it's endless * move packaging files * add packaging files * modify packaging paths * remove redundant files * remove redundant import * fix stimulus set / assembly link * fix stimulus set / assembly indexing * add image label to threshold elevation calculation * change stimulus numbers in the test to be what they should be * add calls to require_variance * fix test errors * fix bug with ceiling access * correct test with incorrect nan droppign * fix wrong number of stimuli in the vernier only test * add comment to explain the logic behind the scoring function * remove redundant comment * remove pool score * add metric tests * fix benchmark filtering issue and recompute ceilings to accommodate removal of pool_score * fix superfluous test by switching it for another * add vernier acuity benchmark * update vernier acuity benchmark, ceilings, and mean ceiling * clean up benchmark file * fix a few bugs with loading benchmarks and such * fix some happy little bugs * add alexnet test * fix dataset argument error * Apply suggestions from code review Co-authored-by: Martin Schrimpf <mschrimpf@users.noreply.github.com> * add renaming suggestions * fix naming issues * remove out-dated aggregation dim in test * fix identifier testing --------- Co-authored-by: Martin Schrimpf <mschrimpf@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR for Psychometric Threshold metric and the Malania2007 benchmarks.
Brief todo: