Create NLDAS2 with full met layers

[rburghol]

Overview

• Note: The workflow steps used are annotated here Workflow met model_meteorology#64 (comment)
• The met workflow for the nldas2 dataset

@COBrogan this is apparently been hard-wired into the program despite our efforts otherwise! I will sort this out today and insure that the values are set properly ... the TS_BAND property is set to false for many of the data sets, which should indicate that all bands will be imported, but, we have zero datasets in the database with all bands, so it seems that this has been the hard-wired for a while. But this should not be too hard to sort out, I am working through it now.

• use varkey = nldas2_obs_hourly
• Final processed file is /backup/meteorology/2025/001/NLDAS_FORA0125_H.A20250101.1800.020.nc.bil_CBP.gtiff
  • This file has only 1 band... this is due to a single line in /opt/model/meta_model/models/raster_met/met/process/01_translate_nc which uses gdalwarp to extract the Rainf bad into a .bil file with a single layer.
  • This seems as if it is a new introduction, but all of the imported rasters thus far, even those that occurred before the shift to .nc have only a single band.

Steps:

• Always keep all layers in .grb file created at the beginning of workflow process step
  • Do not overwrite a .grb file if it exists (huge processing time) and add a switch to allow force overwrite
• Create separate step for tiff creation
• Create separate step for layer extraction if required
• Create separate step for clip
• test
  • test multilayer is kept in final tiff run_model raster_met nldas2all 2025-01-01 auto met process
  • test single later is kept in final tiff run_model raster_met nldas2 2025-01-01 auto met process
• clean up temp file names to be set once in each script and insure cleanup after
• remove old bil related files nohup ./cleanbil &
  • deq3
  • deq2
• Test final import
  • Do imports from 2020-2025 (see comment here for code: Workflow met model_meteorology#64 (comment) )
  • Do imports from 1984-2019
    • since these were different source, we will test whether the process is seamless
    • Note: verify that the convert nc workflow ONLY triggers if there are NC files and does not delete existing grb files otherwise
• rsync from deq3 to deq2 /backup/meteorology
  • Work with Denton to decide on best place to share data in /backup/meteorology so no need for future rsync
• Create new met2date thru 2025
  • Handle new file naming convention?
    • Can not find: /backup/meteorology/2023/002/NLDAS_FORA0125_H.A20230102.0000.002.grb
    • New file name: /backup/meteorology/2023/002/NLDAS_FORA0125_H.A20230102.0000.020.grb
    • See below rename_nldas_020_002
• Update meta_model code with new steps
• Do drought mash up
  • add as step in wdm work flow
• Develop full met csv creation from postgis
  • can we create a temp raster of the met raster clip, then call successive exports?

Scripts

Clean up old .bil files with ./cleanbil

#!/bin/bash
for year in 2020 2021 2022 2023 2024 2025;do
  echo $year
  ndays=`date -d "${year}-12-31" '+%j'`
  # or, if in the middle of a year
  #ndays=`date '+%j'`
  sday=0
  ndays=$((ndays - 1))
  for j in `seq $sday $ndays`; do
    jday=`date -d "${year}-01-01 $j days" '+%j'`
    mdate=`date -d "${year}-01-01 $j days" '+%Y-%m-%d'`
    echo "cd /backup/meteorology/$year/$jday"
    cd /backup/meteorology/$year/$jday
    for i in `ls | grep bil`; do rm $i; done
    for i in `ls | grep prj`; do rm $i; done
    for i in `ls | grep hdr`; do rm $i; done
    for i in `ls | grep 020.tiff`; do rm $i; done
  done
done

Rename grb files from 020 to 002 saved here: https://github.com/HARPgroup/model_meteorology/blob/main/sh/rename_nldas_020_002

#!/bin/bash
basedir="/backup/meteorology"
#for year in 2020 2021 2022 2023 2024 2025;do
for year in 2023;do
  echo $year
  ndays=`date -d "${year}-12-31" '+%j'`
  # or, if in the middle of a year
  #ndays=`date '+%j'`
  sday=0
  ndays=$((ndays - 1))
  for j in `seq $sday $ndays`; do
    jday=`date -d "${year}-01-01 $j days" '+%j'`
    cd $basedir/$year/$jday
    mdate=`date -d "${year}-01-01 $j days" '+%Y-%m-%d'`
    ofiles=`ls *.grb`
    for ofile in $ofiles; do
      file_base=${ofile%.*}
      file_base=${file_base%.*}
      new_file="${file_base}.002.grb"
      echo "mv $ofile $new_file"
      mv $ofile $new_file
    done
  done
done

Rerun Process conversion step to grb

scenario="nldas2all"
for year in 2020 2021 2022 2023 2024 2025; do
   sday=1;
   ndays=`date -d "${year}-12-31" '+%j'`;
   ndays=$((ndays - 1));
   for i in `seq $sday $ndays`; do
     job_name=`cbp mm_job_name $scenario $i`;
     mdate=`date -d "${year}-01-01 $i days" '+%Y-%m-%d'`;
     echo "sbatch --job-name=$job_name run_model raster_met $scenario "$mdate" auto met process 01"; 
     sbatch --job-name=$job_name run_model raster_met $scenario "$mdate" auto met process 01; 
  done; 
done

Convert Kelvin to Celsius

cp /backup/meteorology/2024/010/NLDAS_FORA0125_H.A20240110.0800.020.grb.repro.gtiff ./
# gdal_calc.py cannot determine file from extension .gtiff
mv NLDAS_FORA0125_H.A20240110.0800.020.grb.repro.gtiff NLDAS_FORA0125_H.A20240110.0800.020.grb.repro.tif

gdal_calc.py -A NLDAS_FORA0125_H.A20240110.0800.020.grb.repro.tif --A_band=1 --calc="A - 273.0" --overwrite --outfile=NLDAS_FORA0125_H.A20240110.0800.020.grb.repro.tif

Validate K to C conversion

# make a copy of the previous non-converted version
cp /backup/meteorology/2023/062/NLDAS_FORA0125_H.A20230303.1400.002.grb ~/
raster2pgsql ~/NLDAS_FORA0125_H.A20230303.1400.002.grb tmp_kelvin > /tmp/kelvin.sql
# run for a single day
sbatch run_model raster_met nldas2all "2023-03-03" auto met process 01
# make SQL table for new version with conversion
raster2pgsql /backup/meteorology/2023/062/NLDAS_FORA0125_H.A20230303.1400.002.grb tmp_celsius  > /tmp/celsius.sql
# add to database:
cat /tmp/celsius.sql | psql -h dbase2 drupal.dh03
cat /tmp/kelvin.sql | psql -h dbase2 drupal.dh03

# go into database and compare results of summary
psql -h dbase2 drupal.dh03
SELECT ST_SummaryStatsAgg(rast, 1, TRUE) from tmp_kelvin;

#### Test `make_met2date`
- `sbatch ./make_met2date N51165 19840101 20250531 "2023 2024 2025" landseg`
- See inventory queries: https://github.com/HARPgroup/model_meteorology/issues/50

[rburghol]

QA. Original (from 2021 to 2025)

 2021 | 2021-01-01 00:00:00-05 | 2021-12-31 23:00:00-05 |  8760 |    1 |     40.44983590066135
 2022 | 2022-01-01 00:00:00-05 | 2022-12-31 23:00:00-05 |  8760 |    1 |     38.81266468851347
 2023 | 2023-01-01 00:00:00-05 | 2023-12-31 23:00:00-05 |  8760 |    1 |     38.31938150664708
 2024 | 2024-01-01 00:00:00-05 | 2024-12-31 23:00:00-05 |  8784 |    1 |     37.67803946677503
 2025 | 2025-01-01 00:00:00-05 | 2025-05-26 08:00:00-04 |  3488 |    1 |     14.90370572505855

New:

 year |       start_date       |        end_date        | count | band |      band_sum
------+------------------------+------------------------+-------+------+--------------------
 2020 | 2020-01-01 19:00:00-05 | 2020-12-31 18:00:00-05 |  8758 |   10 |  42.62629531313033
 2021 | 2021-01-01 19:00:00-05 | 2021-12-31 18:00:00-05 |  8735 |   10 |  37.88007776487938
 2022 | 2022-01-01 19:00:00-05 | 2022-12-31 18:00:00-05 |  8733 |   10 |  38.17436273164002
 2023 | 2023-01-01 19:00:00-05 | 2023-12-31 18:00:00-05 |  8734 |   10 | 38.096274310732056
 2024 | 2024-01-01 19:00:00-05 | 2024-12-31 23:00:00-05 |  8764 |   10 |  39.37536157810769
 2025 | 2025-01-01 00:00:00-05 | 2025-06-28 08:00:00-04 |  4279 |   10 | 20.337007018993166
(6 rows)