Merge dimension mashup

[bblay]

🐛 Bug Report

Merge is mashing two dimensions into one.

How To Reproduce

I didn't manage to reproduce this using contrived cubes, so I can only currently reproduce it from the source data, at work. Simply load the cubes and the problem occurs. I've written a script which demonstrates the problem and will paste that, and the output, below. I also have a workaround at work. I'm not yet sure if I can share it.

Expected behaviour

The data are global lat-lon octants. I believe they should be loaded into sixteen cubes. Eight of them should share a single time and forecast period, with fourteen pressure levels. The other eight should share a different time and forecast period, and have 11 pressure levels.

Actual behaviour

Instead, the data is loading into eight cubes with the third dimension having 25 points - a mashup of the pressure and time coords.

One impact of this: I'm extracting each time point and saving as netcdf. Without the workaround, it produces a file without proper knowledge of the pressure dimension. It's just called "dim1", it's in a seemingly random order, and the points are just indices 0-13. Another impact is that the coord points are a bit of a mess, as you can see in the output below.

Workaround

I've worked around this by using load_raw, and writing a function which groups by time, merging each group separately to force the pressure levels to be merged into a sensible dimension first. The workaround is specific to the data, not a generic fix.

Environment

• OS & Version: RHEL 7.9
• Iris Version: 3.7

Additional context

code to highlight the problem

#!/usr/bin/env python

from pathlib import Path
import sys
sys.path.append('HIDING_PATHS_HERE')

import iris
from iris.cube import CubeList

# workaround is in here
from grib_concat import load_cubes, load_cubes_controlled


if __name__ == '__main__':

    fpaths = [
        Path('HIDING_PATHS_HERE/file1'),
        Path('HIDING_PATHS_HERE/file2'),
    ]

    # normal iris loading
    iris_load_cubes = load_cubes(fpaths)
    iris_156_cubes = CubeList([c for c in iris_load_cubes if 'UNKNOWN LOCAL PARAM 156' in c.name()])

    # workaround with controlled merging
    controlled_merge_cubes = load_cubes_controlled(fpaths)
    controlled_156_cubes = CubeList([c for c in controlled_merge_cubes if 'UNKNOWN LOCAL PARAM 156' in c.name()])

    print('--- iris load ---\n')

    print(f'{len(iris_156_cubes)} cubes\n')
    for c in iris_156_cubes:
        print(repr(c))

    print('\ncube 0:\n', iris_156_cubes[0], '\n')
    print('forecast_period', iris_156_cubes[0].coord('forecast_period').points)
    print('pressure', iris_156_cubes[0].coord('pressure').points)
    print('time', iris_156_cubes[0].coord('time').points, '\n')

    print('one of the cubes after extracting fp=0:')
    iris_fp0 = iris_156_cubes.extract(iris.Constraint(forecast_period=0))[0]
    print(repr(iris_fp0))
    print('forecast_period', iris_fp0.coord('forecast_period').points)
    print('pressure', iris_fp0.coord('pressure').points)
    print('time', iris_fp0.coord('time').points, '\n')
    iris.save(iris_fp0, 'from_iris_load.nc')

    print('--- controlled merge ---\n')

    print(f'{len(controlled_156_cubes)} cubes\n')
    for c in controlled_156_cubes:
        print(repr(c))

    print('\ncube 0:\n', controlled_156_cubes[0], '\n')
    print('forecast_period', controlled_156_cubes[0].coord('forecast_period').points)
    print('pressure', controlled_156_cubes[0].coord('pressure').points)
    print('time', controlled_156_cubes[0].coord('time').points, '\n')

    print('one of the cubes after extracting fp=0:')
    controlled_fp0 = controlled_156_cubes.extract(iris.Constraint(forecast_period=0))[0]
    print(repr(controlled_fp0))
    print('forecast_period', controlled_fp0.coord('forecast_period').points)
    print('pressure', controlled_fp0.coord('pressure').points)
    print('time', controlled_fp0.coord('time').points, '\n')
    iris.save(controlled_fp0, 'from_controlled_merge.nc')

In the output below, you can first see the data loaded with iris.load, then using iris.load_raw and "controlled" merging. In both cases one cube is printed, and then cubes are extracted at forecast_period=0 with one of those printed.

output

--- iris load ---

8 cubes

<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>

cube 0:
 UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)
    Dimension coordinates:
        latitude                            -             x              -
        longitude                           -             -              x
    Auxiliary coordinates:
        forecast_period                     x             -              -
        pressure                            x             -              -
        time                                x             -              -
    Scalar coordinates:
        originating_centre              European Centre for Medium Range Weather Forecasts
    Attributes:
        generatingProcessIdentifier     154
        indicatorOfParameter            156
        indicatorOfTypeOfLevel          'pl'
        originating_centre_id           'ecmf' 

forecast_period [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0 24 24 24 24 24 24 24 24 24 24
 24]
pressure [ 300  250  400 1000  700  925  200  850  500  100  150   50   70   30
  700  925  300  200  850 1000  100  400  500  250  150]
time [474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240.
 474240. 474240. 474240. 474240. 474240. 474264. 474264. 474264. 474264.
 474264. 474264. 474264. 474264. 474264. 474264. 474264.] 

one of the cubes after extracting fp=0:
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 14; latitude: 61; longitude: 60)>
forecast_period [0 0 0 0 0 0 0 0 0 0 0 0 0 0]
pressure [ 300  250  400 1000  700  925  200  850  500  100  150   50   70   30]
time [474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240.
 474240. 474240. 474240. 474240. 474240.] 

--- controlled merge ---

16 cubes

<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>

cube 0:
 UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)
    Dimension coordinates:
        pressure                                 x             -              -
        latitude                                 -             x              -
        longitude                                -             -              x
    Scalar coordinates:
        forecast_period                 0 hours
        originating_centre              European Centre for Medium Range Weather Forecasts
        time                            2024-02-07 00:00:00
    Attributes:
        generatingProcessIdentifier     154
        indicatorOfParameter            156
        indicatorOfTypeOfLevel          'pl'
        originating_centre_id           'ecmf' 

forecast_period [0]
pressure [  30   50   70  100  150  200  250  300  400  500  700  850  925 1000]
time [474240.] 

one of the cubes after extracting fp=0:
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
forecast_period [0]
pressure [  30   50   70  100  150  200  250  300  400  500  700  850  925 1000]
time [474240.] 

[github-actions]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.