[Question] Ensemble approach using multiple runs from checkpoint files

[Khreat0205]

Background

I'm currently using scTab for cell type annotation on my research dataset with the checkpoint file cap-sctab-service-ckpts.tar.gz downloaded from here.

Current Approach

The checkpoint appears to contain 5 runs with different random seeds. I'm implementing an ensemble approach as follows:

1. Prediction aggregation: Use all 5 runs to predict cell types for each cell
2. Label assignment: Assign the most frequent (mode) predicted label across the 5 runs
3. Tie-breaking: If there's a tie in the mode calculation, compute the average prediction probability for each tied label and assign the label with higher average probability
4. Confidence scoring: Use the average probability from runs that predicted the assigned label as the final confidence score
5. Filtering: Plan to filter cells based on this confidence score

Questions

1. Is this ensemble approach appropriate and recommended? Or would it be better to use a specific single run from the checkpoint?
2. Are there any best practices or recommendations for handling multiple runs in scTab checkpoints?
3. Is the tie-breaking method reasonable, or should I consider alternative approaches (e.g., using prediction entropy, maximum probability across runs, etc.)?
4. For confidence-based filtering, what threshold ranges have you found effective in practice?

Any guidance on the optimal strategy for utilizing multiple runs would be greatly appreciated!

[felix0097]

Hi @Khreat0205,

What you usually do is that you take the average across the predicted probabilities of the individual models. Based on this "averaged probability vector", you can do then do cell-type assignments (argmax) and estimate the uncertainty e.g. 1 - max_probability or the entropy of the probability vector (you can check which works best for you, in the paper we just used 1 - max_prob).

In terms of which threshold to use, this is usually dataset dependent. What you could do is overlay the uncertainty estimates over e.g. a UMAP plot. This way you get an overview, which cells are associated with a particular high uncertainty.

Another remark, for using scTab for "fully automated" cell annotation. The problem is that the CELLxGENE training data is pretty flawed (the ontology matching they use to "harmonize" / align cell-type labels across individual datasets is pretty flawed and doesn't really work in practice). Hence, you have pretty pure label quality which often leads to poor model performance (especially in a setting where you have completely novel datasets). So take the predictions with a grain of salt and always double check the predictions manually (we're unfortunately still pretty far a way from fully automated cell-type classification).