Fix a prototype homgraphy view

[mosmar99]

Currently there is no model or tool we have to map from video input to a seperate top-down view of player positions on court. Todo this we need to identify key points on the map and the player positions in relations to these points. If we could identify the ball position aswell that would be useful but it can be harder since the ball is not always on the ground, it bounding box floates above it and its not yet clear how to estimate its position from a topdown view based on that. The same problem applies to players when they are jumping. Nonetheless, a first problem is simply to generally identify players on the map.

Please, write some tests aswell.

[isacpaulsson]

YOLO8 and YOLO11 keypoint detection (Not working)

Court homography using keypoint detection with YOLO8 and YOLO11 using two different datasets (1, 2). Performance has been sub-par, generally losing tracking on center of court. Additionally, other projects have utilized the method and tend to only show example videos of courts that exist in the dataset. Models show poor generalization, attempts to reproduce result using both provided and our fine tuned models to data from without the dataset.

Example keypoint detection (Frame points window shows the keypoint detections in the video, Ref points shows the corresponding detected keypoints in the reference, the Warped image window shows the video reprojected using homography generated from the keypoints (Here we can imagine the reprojected player coordinates that would result from the homography)):

2025-11-12.12-59-16.mp4

When facing either end of the court, keypoints are decent. Rule based filtering based on the proportions of the reference keypoints could be used to filter out stray matches, this has not been implemented due to the lack of tracking points on the center of the court.

SIFT based court homography (Works for panorama)

Initial SIFT based experiments were based on simplified reference images of the court


SIFT and similar methods rely heavily on image texture which these images lack. This method was unable to find reasonable homographies.

To improve the matching, a reference image of the actual court could help. Since these are not generally available, one can be generated from a reference video similarly to a panorama. Using SIFT on consecutive frames in a video provides many suitable matches allowing a panorama to be built. To account for perspective warp on camera rotation, a cylindrical reprojection is applied to the images from the video. To reduce the number of frames to be stitched intermediate frames can be dropped. To filter out players the typical value can be used, This is currently implemented using median, this could be improved using the actual typical value possibly after binning.
Example resulting panorama:

Given the (x, y) coordinates of the corner points of the court in the image a homography matrix can be generated to reproject the panorama yeilding our final reference image more suitable for matching.

Matching the reference image to the video frames using SIFT still does not provide a viable solution but is able to find reasonable homographies for approximately 50% of the frames.

SuperPoint and LightGlue (Best so far for court homography)

SuperPoint and LightGlue are deep learning based alternatives to SIFT and bfmatcher (from cv2). Swapping to these models provides a large improvement to performance when matching to video frame to reference:

2025-11-12.13-35-32.mp4

Ideas for improvement

Move from median to typical value in court stitching.

Currently requires user input for marking corners in panorama, could possibly be automated. Yolo8/11 keypoint detection fails here, both on full image and split image (Reason unknown). Another alternative may be to use sam2/3. Bellow is an exemple prediction with sam3 and prompt basketball court. The mask paired with line detection (e.g., hough) could be viable.

Currently focal length in cylindrical reprojection is based on image width. A better estimate would likely result in a better reprojection and panorama more robust to variance between cameras.