Skip to content

Changes in chroma SATD calc for YCBCR422, potential overflow of the cost accumulator #223

Description

@gakssultang

Issue 1 - Incorrect chroma SATD for YCBCR422

The tile->rc.cost computed by v0.3.0.0 for a YCBCR422 input differs from v0.2.1.0 (using the fastest rate-control preset). Is this change intended?

/* oapve_rc_get_tile_cost */
int tx = (tile->x + x) >> ctx->c_sft[c][0];
int ty = (tile->y + y) >> ctx->c_sft[c][1];
...
ctx->fn_blk_from_pic[c](OAPV_BLK_W, OAPV_BLK_H, pic, tx, org_s,
                        core->coef, (OAPV_BLK_W << 1), ctx->bit_depth, 0);

With the fix above, I was able to obtain the same result as v0.2.1.0.

Issue 2 - Potential overflow in cost accumulator

The cost accumulator (sum) is currently declared as a 32-bit signed int. For larger tiles (e.g., a whole 8K frame as one tile), higher bit depth, higher-energy content, the accumulation may exceed INT_MAX.

Note the destination tile->rc.cost is already a double (see oapve_rc_tile_t in src/oapv_def.h), so widening the accumulator has no precision downside. The related aggregate oapve_rc_get_tile_cost_thread() already uses a u64* sum across tiles — the per-tile accumulator is the one still on int.

/* oapve_rc_get_tile_cost */
double sum = 0;
...
tile->rc.cost = sum;

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions