Question about reproducing PA-MPJPE results with detected bounding boxes

[louzihan9]

Hi, thank you for your excellent work.

While reading your paper, I noticed that you reported a PA-MPJPE of 26.9 for the 3DB baseline and 29.7 for Fast3DB when using detected bounding boxes. However, in my reproduction, I was only able to obtain a PA-MPJPE of around 36 for the 3DB baseline.

Would you be willing to share some additional evaluation details that could help me reproduce this part of the results? For example, any specific settings, preprocessing steps, detected bounding boxes, checkpoints, or evaluation scripts used in your experiments would be very helpful.

Thank you very much for your time and help!

[yangtiming]

Hi,

Quick question: are you computing PA-MPJPE with the 24-joint set or the 14-joint set?

If you can share your eval code, I'll try my best to check where it diverges. We've open-sourced everything except the evaluation scripts — those differ per dataset and weren't the focus of this work.

Thanks!

[louzihan9]

Thanks for your reply!

We compute PA-MPJPE using the 14-joint set.

I’m also happy to share our evaluation code. Since the data storage format on my server may be different from yours, I used an AI tool to help me extract and clean up the core metric computation part of our evaluation script. This code is mainly adapted from the evaluation logic used in GVHMR.

The code is shown below:

"""Standalone GVHMR-style 3DPW metric core.

This file is a compact, self-contained version of:

- projects/evaluate_3dpw/metric_core.py
- projects/evaluate_3dpw/gvhmr_standard.py

It is intended for sharing and discussion of the metric protocol, not as a full
benchmark runner. In particular, this file does not implement:

- loading 3DPW files;
- gendered SMPL forward for GT;
- GVHMR supermotion_EVAL3DPW prediction forward;
- smplx2smpl_sparse.pt conversion;
- 3DPW-14 joint regression.

Those steps should be performed by the caller. The public entry point here is
``evaluate_gvhmr_standard(...)``.

Expected metric-ready inputs:

- ``gt_vertices_smpl``:
  GT SMPL vertices in camera coordinates, SMPL topology, meters.
  In our benchmark this is produced from gendered 3DPW SMPL, then transformed
  by 3DPW world_to_camera.

- ``gt_joints_3dpw14``:
  GT 3DPW-14 joints in camera coordinates, meters, from the benchmark sparse
  joint regressor.

- ``pred_vertices_smpl``:
  Prediction vertices in camera coordinates, converted to SMPL topology.
  In the GVHMR benchmark path this means the prediction has already gone
  through supermotion_EVAL3DPW and smplx2smpl_sparse.pt.

- ``pred_joints_3dpw14``:
  Predicted 3DPW-14 joints in camera coordinates, meters, from the same sparse
  joint regressor.

- ``valid``:
  Optional bool mask with the same sample shape as the inputs. For a sequence
  this is typically ``(T,)`` or ``(T, P)``. GVHMR acceleration first selects
  valid frames, then computes the second difference on the compressed sequence.

Supported input shapes:

- Single sample: ``(V, 3)`` vertices and ``(14, 3)`` joints.
- Sequence: ``(T, V, 3)`` and ``(T, 14, 3)``.
- Multi-person sequence: ``(T, P, V, 3)`` and ``(T, P, 14, 3)``.

Output units:

- PA-MPJPE, MPJPE, PVE: millimeters.
- Accel: meters / second^2.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Sequence

import numpy as np


RootSpec = int | Sequence[int]
GVHMR_PELVIS_IDXS = (2, 3)
GVHMR_PROTOCOL = (
    "gvhmr-official: 3DPW-14 joints, pelvis mean [2,3] root, "
    "SMPL topology after supermotion_EVAL3DPW + smplx2smpl_sparse, "
    "mask-compressed sequence acceleration"
)


@dataclass(frozen=True)
class MetricResult:
    """Aggregated metric values."""

    protocol: str
    pa_mpjpe: float
    mpjpe: float
    pve: float
    accel: float
    counts: dict[str, int]
    units: dict[str, str] = field(
        default_factory=lambda: {
            "pa_mpjpe": "mm",
            "mpjpe": "mm",
            "pve": "mm",
            "accel": "m/s^2",
        }
    )

    def as_dict(self) -> dict[str, object]:
        return {
            "protocol": self.protocol,
            "metrics": {
                "pa_mpjpe": self.pa_mpjpe,
                "mpjpe": self.mpjpe,
                "pve": self.pve,
                "accel": self.accel,
            },
            "counts": dict(self.counts),
            "units": dict(self.units),
        }


def evaluate_gvhmr_standard(
    *,
    gt_vertices_smpl: np.ndarray,
    gt_joints_3dpw14: np.ndarray,
    pred_vertices_smpl: np.ndarray,
    pred_joints_3dpw14: np.ndarray,
    valid: np.ndarray | None = None,
    fps: float = 30.0,
) -> MetricResult:
    """Compute GVHMR/benchmark-style 3DPW metrics.

    Protocol details:

    1. Flatten all leading sample dimensions while preserving ``sample_shape``.
    2. Require matching SMPL vertex topology for PVE.
    3. Require exactly 14 joints for both GT and prediction.
    4. Root-align joints and vertices by the mean of 3DPW-14 hip joints
       ``[2, 3]``.
    5. Mask invalid samples before metric aggregation.
    6. MPJPE is mean joint error after pelvis-root subtraction.
    7. PA-MPJPE is mean joint error after per-sample similarity Procrustes
       alignment of predicted rooted 3DPW-14 joints to GT rooted 3DPW-14 joints.
    8. PVE is mean vertex error after the same pelvis-root subtraction.
    9. Acceleration follows GVHMR behavior: valid frames are selected first,
       then second-order finite differences are computed on that compressed
       sequence for each person track independently.
    """

    arrays = _prepare_gvhmr_arrays(
        gt_vertices_smpl=gt_vertices_smpl,
        gt_joints_3dpw14=gt_joints_3dpw14,
        pred_vertices_smpl=pred_vertices_smpl,
        pred_joints_3dpw14=pred_joints_3dpw14,
        valid=valid,
    )
    gt_v = arrays["gt_vertices"]
    gt_j = arrays["gt_joints"]
    pred_v = arrays["pred_vertices"]
    pred_j = arrays["pred_joints"]
    keep = arrays["valid"]
    sample_shape = arrays["sample_shape"]

    gt_pelvis = joint_root(gt_j, GVHMR_PELVIS_IDXS)
    pred_pelvis = joint_root(pred_j, GVHMR_PELVIS_IDXS)
    gt_j_rooted = gt_j - gt_pelvis
    pred_j_rooted = pred_j - pred_pelvis
    gt_v_rooted = gt_v - gt_pelvis
    pred_v_rooted = pred_v - pred_pelvis

    finite = (
        np.isfinite(gt_j_rooted).all(axis=(1, 2))
        & np.isfinite(pred_j_rooted).all(axis=(1, 2))
        & np.isfinite(gt_v_rooted).all(axis=(1, 2))
        & np.isfinite(pred_v_rooted).all(axis=(1, 2))
    )
    keep = keep & finite

    if not np.any(keep):
        return MetricResult(
            protocol=GVHMR_PROTOCOL,
            pa_mpjpe=np.nan,
            mpjpe=np.nan,
            pve=np.nan,
            accel=np.nan,
            counts={
                "sample_count": 0,
                "joint_count": 0,
                "vertex_count": 0,
                "accel_count": 0,
            },
        )

    pred_j_keep = pred_j_rooted[keep]
    gt_j_keep = gt_j_rooted[keep]
    pred_v_keep = pred_v_rooted[keep]
    gt_v_keep = gt_v_rooted[keep]

    pred_pa = similarity_align_batch(pred_j_keep, gt_j_keep)
    pa_per_sample_mm = np.linalg.norm(pred_pa - gt_j_keep, axis=-1).mean(axis=-1) * 1000.0
    mpjpe_per_sample_mm = np.linalg.norm(pred_j_keep - gt_j_keep, axis=-1).mean(axis=-1) * 1000.0
    pve_per_sample_mm = np.linalg.norm(pred_v_keep - gt_v_keep, axis=-1).mean(axis=-1) * 1000.0
    accel_per_sample = gvhmr_mask_compressed_accel(
        gt_j_rooted=gt_j_rooted,
        pred_j_rooted=pred_j_rooted,
        valid=keep,
        sample_shape=sample_shape,
        fps=fps,
    )

    return MetricResult(
        protocol=GVHMR_PROTOCOL,
        pa_mpjpe=float(np.mean(pa_per_sample_mm)),
        mpjpe=float(np.mean(mpjpe_per_sample_mm)),
        pve=float(np.mean(pve_per_sample_mm)),
        accel=float(np.mean(accel_per_sample)) if accel_per_sample.size else np.nan,
        counts={
            "sample_count": int(pred_j_keep.shape[0]),
            "joint_count": int(pred_j_keep.shape[0] * pred_j_keep.shape[1]),
            "vertex_count": int(pred_v_keep.shape[0] * pred_v_keep.shape[1]),
            "accel_count": int(accel_per_sample.size),
        },
    )


def prepare_metric_ready_inputs(
    *,
    gt_smpl_params: object | None = None,
    pred_gvhmr_outputs: object | None = None,
    world_to_camera: np.ndarray | None = None,
    smplx2smpl_sparse_path: str | None = None,
    joint_regressor_3dpw14: object | None = None,
) -> dict[str, np.ndarray]:
    """Placeholder interface for code outside this file.

    This function intentionally raises ``NotImplementedError``. It documents the
    boundary between model/data preparation and metric computation.

    A real implementation should return a dict containing:

    - ``gt_vertices_smpl``
    - ``gt_joints_3dpw14``
    - ``pred_vertices_smpl``
    - ``pred_joints_3dpw14``
    - optional ``valid``

    In our benchmark, the missing implementation performs:

    1. gendered 3DPW SMPL forward for GT in world coordinates;
    2. transform GT vertices/joints by 3DPW ``world_to_camera``;
    3. GVHMR prediction forward with ``supermotion_EVAL3DPW``;
    4. sparse SMPL-X-to-SMPL topology conversion via ``smplx2smpl_sparse.pt``;
    5. sparse 3DPW-14 joint regression for both GT and prediction.
    """

    raise NotImplementedError("Provide metric-ready arrays, then call evaluate_gvhmr_standard(...).")


def apply_world_to_camera(points: np.ndarray, world_to_camera: np.ndarray) -> np.ndarray:
    """Apply batched 4x4 world-to-camera transforms to points.

    ``points`` can be ``(..., N, 3)``. ``world_to_camera`` can be ``(4, 4)`` or
    any broadcast-compatible ``(..., 4, 4)`` transform array.
    """

    pts = np.asarray(points, dtype=np.float32)
    transforms = np.asarray(world_to_camera, dtype=np.float32)
    if pts.shape[-1] != 3:
        raise ValueError(f"points must end with 3 coordinates, got {pts.shape}")
    if transforms.shape[-2:] != (4, 4):
        raise ValueError(f"world_to_camera must end with (4,4), got {transforms.shape}")

    point_batch = pts.shape[:-2]
    transform_batch = transforms.shape[:-2]
    target_batch = np.broadcast_shapes(point_batch, transform_batch)

    pts_b = np.broadcast_to(pts, target_batch + pts.shape[-2:])
    tr_b = np.broadcast_to(transforms, target_batch + (4, 4))
    rotation = tr_b[..., :3, :3]
    translation = tr_b[..., :3, 3]
    return (np.einsum("...ij,...nj->...ni", rotation, pts_b) + translation[..., None, :]).astype(np.float32)


def aggregate_results(protocol: str, results: Sequence[MetricResult]) -> MetricResult:
    """Merge scene-level results with the same weighting used in our jobs."""

    sample_count = sum(r.counts.get("sample_count", 0) for r in results)
    joint_count = sum(r.counts.get("joint_count", 0) for r in results)
    vertex_count = sum(r.counts.get("vertex_count", 0) for r in results)
    accel_count = sum(r.counts.get("accel_count", 0) for r in results)

    def weighted(metric_name: str, count_name: str) -> float:
        denom = sum(r.counts.get(count_name, 0) for r in results)
        if denom == 0:
            return np.nan
        total = 0.0
        for result in results:
            value = getattr(result, metric_name)
            count = result.counts.get(count_name, 0)
            if count and np.isfinite(value):
                total += value * count
        return float(total / denom)

    return MetricResult(
        protocol=protocol,
        pa_mpjpe=weighted("pa_mpjpe", "sample_count"),
        mpjpe=weighted("mpjpe", "joint_count"),
        pve=weighted("pve", "vertex_count"),
        accel=weighted("accel", "accel_count"),
        counts={
            "sample_count": int(sample_count),
            "joint_count": int(joint_count),
            "vertex_count": int(vertex_count),
            "accel_count": int(accel_count),
        },
    )


def joint_root(joints: np.ndarray, root: RootSpec) -> np.ndarray:
    """Return one root point per sample with shape ``(N, 1, 3)``."""

    if isinstance(root, int):
        return joints[:, root : root + 1]
    root_indices = np.asarray(list(root), dtype=np.int64)
    if root_indices.size == 0:
        raise ValueError("root index list cannot be empty")
    return joints[:, root_indices].mean(axis=1, keepdims=True)


def similarity_align_batch(source: np.ndarray, target: np.ndarray) -> np.ndarray:
    """Per-sample similarity Procrustes alignment.

    ``source`` and ``target`` must both be ``(N, J, 3)``. The returned array has
    the same shape as ``source`` and is aligned to ``target``.
    """

    source = np.asarray(source, dtype=np.float32)
    target = np.asarray(target, dtype=np.float32)
    if source.shape != target.shape or source.ndim != 3 or source.shape[-1] != 3:
        raise ValueError(f"expected matching (N,J,3), got {source.shape} and {target.shape}")

    mu_source = source.mean(axis=1, keepdims=True)
    mu_target = target.mean(axis=1, keepdims=True)
    source0 = source - mu_source
    target0 = target - mu_target
    var_source = np.sum(source0 * source0, axis=(1, 2))
    aligned = np.full_like(source, np.nan, dtype=np.float32)
    safe = var_source > 1e-12
    if not np.any(safe):
        return aligned

    covariance = np.matmul(source0[safe].transpose(0, 2, 1), target0[safe])
    u, singular_values, vt = np.linalg.svd(covariance)
    z = np.tile(np.eye(3, dtype=np.float32), (u.shape[0], 1, 1))
    z[:, -1, -1] = np.sign(np.linalg.det(np.matmul(u, vt)))
    rotation = np.matmul(np.matmul(u, z), vt)
    scale = np.sum(singular_values * np.diagonal(z, axis1=1, axis2=2), axis=1) / var_source[safe]
    transformed = scale[:, None, None] * np.matmul(source[safe], rotation)
    translation = mu_target[safe] - scale[:, None, None] * np.matmul(mu_source[safe], rotation)
    aligned[safe] = transformed + translation
    return aligned


def gvhmr_mask_compressed_accel(
    *,
    gt_j_rooted: np.ndarray,
    pred_j_rooted: np.ndarray,
    valid: np.ndarray,
    sample_shape: tuple[int, ...],
    fps: float,
) -> np.ndarray:
    """GVHMR acceleration: mask first, then differentiate.

    For a multi-person sequence, every person track is handled independently.
    If a track has fewer than three valid frames after masking, it contributes
    no acceleration samples.
    """

    if len(sample_shape) == 0 or sample_shape[0] < 3:
        return np.zeros((0,), dtype=np.float32)

    num_joints = gt_j_rooted.shape[-2]
    gt_seq = gt_j_rooted.reshape(sample_shape + (num_joints, 3))
    pred_seq = pred_j_rooted.reshape(sample_shape + (num_joints, 3))
    valid_seq = valid.reshape(sample_shape)

    if len(sample_shape) == 1:
        gt_tracks = gt_seq[:, None]
        pred_tracks = pred_seq[:, None]
        valid_tracks = valid_seq[:, None]
    else:
        t = sample_shape[0]
        rest = int(np.prod(sample_shape[1:]))
        gt_tracks = gt_seq.reshape((t, rest, num_joints, 3))
        pred_tracks = pred_seq.reshape((t, rest, num_joints, 3))
        valid_tracks = valid_seq.reshape((t, rest))

    errors: list[np.ndarray] = []
    scale = float(fps) * float(fps)
    for track_idx in range(gt_tracks.shape[1]):
        mask = valid_tracks[:, track_idx]
        if int(mask.sum()) < 3:
            continue
        gt_valid = gt_tracks[mask, track_idx]
        pred_valid = pred_tracks[mask, track_idx]
        gt_acc = gt_valid[:-2] - 2.0 * gt_valid[1:-1] + gt_valid[2:]
        pred_acc = pred_valid[:-2] - 2.0 * pred_valid[1:-1] + pred_valid[2:]
        errors.append(np.linalg.norm(pred_acc - gt_acc, axis=-1).mean(axis=-1) * scale)

    if not errors:
        return np.zeros((0,), dtype=np.float32)
    return np.concatenate(errors).astype(np.float32)


def _prepare_gvhmr_arrays(
    *,
    gt_vertices_smpl: np.ndarray,
    gt_joints_3dpw14: np.ndarray,
    pred_vertices_smpl: np.ndarray,
    pred_joints_3dpw14: np.ndarray,
    valid: np.ndarray | None,
) -> dict[str, np.ndarray | tuple[int, ...]]:
    gt_v, sample_shape = _flatten_samples(gt_vertices_smpl, "gt_vertices_smpl")
    gt_j, sample_shape_gt_j = _flatten_samples(gt_joints_3dpw14, "gt_joints_3dpw14")
    pred_v, sample_shape_pred_v = _flatten_samples(pred_vertices_smpl, "pred_vertices_smpl")
    pred_j, sample_shape_pred_j = _flatten_samples(pred_joints_3dpw14, "pred_joints_3dpw14")

    if not (sample_shape == sample_shape_gt_j == sample_shape_pred_v == sample_shape_pred_j):
        raise ValueError(
            "all GVHMR metric arrays must have the same sample shape; got "
            f"{sample_shape}, {sample_shape_gt_j}, {sample_shape_pred_v}, {sample_shape_pred_j}"
        )
    if gt_v.shape[1:] != pred_v.shape[1:]:
        raise ValueError(f"PVE requires matching SMPL topology, got {gt_v.shape} and {pred_v.shape}")
    if gt_j.shape[1] != 14 or pred_j.shape[1] != 14:
        raise ValueError(f"GVHMR benchmark metrics require exactly 14 3DPW joints, got {gt_j.shape} and {pred_j.shape}")

    if valid is None:
        flat_valid = np.ones((gt_v.shape[0],), dtype=bool)
    else:
        valid_arr = np.asarray(valid, dtype=bool)
        if sample_shape == ():
            flat_valid = np.asarray([bool(valid_arr.item())], dtype=bool)
        else:
            if valid_arr.shape != sample_shape:
                raise ValueError(f"valid mask shape {valid_arr.shape} does not match sample shape {sample_shape}")
            flat_valid = valid_arr.reshape(-1)

    return {
        "gt_vertices": gt_v,
        "gt_joints": gt_j,
        "pred_vertices": pred_v,
        "pred_joints": pred_j,
        "valid": flat_valid,
        "sample_shape": sample_shape,
    }


def _flatten_samples(array: np.ndarray, name: str) -> tuple[np.ndarray, tuple[int, ...]]:
    arr = np.asarray(array, dtype=np.float32)
    if arr.ndim < 2 or arr.shape[-1] != 3:
        raise ValueError(f"{name} must have shape (..., N, 3), got {arr.shape}")
    if arr.ndim == 2:
        return arr[None], ()
    return arr.reshape((-1,) + arr.shape[-2:]), arr.shape[:-2]

I hope this snippet is sufficient to illustrate the evaluation function we are using. Please let me know if you need any additional context or details.

[yangtiming]

It looks good. It looks very similar to our evaluation protocol. Sorry, I don't see anything that would explain the discrepancy.