This repository extends the standard COCO person keypoint evaluation by implementing the Extended-OKS (Ex-OKS) metric introduced in the ProbPose paper. Built on top of the original xtcocotools and the official COCO API, Ex-OKS remains fully backward-compatible with the standard OKS. It adds support for:
- Out-of-image keypoints (points annotated outside the image boundary or activation window) to asses model's robustness
- Per-visibility-level mAP breakdowns to pinpoint which keypoints cause errors
Why ExOCocoTools? ExO stands both for Ex-OKS and for greek "exo", meaning "outside". This evaluation protocol is the first to evaluate on keypoints outside of the image.
- OKS (Object Keypoint Similarity) measures similarity between predicted and ground-truth keypoints within the image.
- Ex-OKS (Extended OKS) extends OKS by:
- Penalizing in-image predictions when the ground-truth is out-of-image
- Penalizing out-of-image predictions when the groud-truth is in-image
- The same as OKS when both ground-truth and prediction are in-image
| Aspect | OKS | Ex-OKS |
|---|---|---|
| What it measures | Alignment of predictions vs. ground-truth using a Gaussian fall-off based on object scale | Same alignment AND corectness of "in-view" vs. "out-of-view" classification |
| Presence handling | Only evaluates localization for keypoints inside the image | Localization for keypoints inside the image and classification for out-of-view keypoints |
| False-positive penalty | None | Penalizes in-image prediction when a keypoint is out-of-view |
| Primary use-case | Benchmark evaluation: Localize all points inside the image | Real-world applications: First predict if the point is "there" and if yes, localize it |
Formally, Ex-OKS has the same form as OKS -- euclidean distance scaled by image scale and per-keypoint sigma.
However, for Ex-OKS, the distance
In summary, Ex-OKS extends OKS to situation when ground-truth or prediction are outside of the activation window. For more details, read the full explanation in the ProbPose paper.
Apart from Ex-OKS, this library also shows mAP for different visibility levels. Best demonstrated in CropCOCO Demo, rows with specific visibility show performance for such keypoints.
The output above shows that for
- occluded keypoints (v=1), the mAP is 38.9
- for visible keypoints (v=2), the mAP is 79.1
- and for out-of-image keypoints (v=3), the mAP is 31.7
from exococotools.coco import COCO
from exococotools.cocoeval import COCOeval
# Standard OKS evaluation (backward-compatible)
cocoEval = COCOeval(cocoGt_json, cocoDt_json, iouType='keypoints', extended_oks=False)
# --- OR ---
# Extended-OKS evaluation
cocoEval = COCOeval(cocoGt_json, cocoDt_json, iouType='keypoints', extended_oks=True)
# Evaluate and print results
cocoEvalExt.evaluate()
cocoEvalExt.accumulate()
cocoEvalExt.summarize()For more details, see COCO Demo or CropCOCO Demo files.
pip install exococotoolsgit clone https://github.com/MiraPurkrabek/Ex-cocotools
cd Ex-cocotools
pip install -r requirements.txt
pip install -e .This implementation builds upon the COCO API and xtcocotools projects. The Extended-OKS metric and its evaluation methodology are described in the ProbPose paper:
@InProceedings{Purkrabek2025CVPR,
author = {Purkrabek, Miroslav and Matas, Jiri},
title = {ProbPose: A Probabilistic Approach to 2D Human Pose Estimation},
booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
month = {June},
year = {2025},
pages = {27124-27133}
}