Elbe use case

[MarkusKonk]

https://github.com/smatmir/aquainfra-usecase-elbe

[MarkusKonk]

@smatmir, @IdaMBonnevie Let's continue the technical discussion here.

[MarkusKonk]

A little update:

• I made some changes to the computational environment. Now the code also runs on MyBinder, except step 2 and 8 which require too much RAM for MyBinder, which only has limited resources.
• Added example results to the repository
• changed the image name for a more meaningful one
• made some minor changes to the code to allow local and remote files

I also created some OGC API Processes already:

And based on these OGC API Processes, I was already able to create a first very rough version of a workflow in my local Galaxy instance:

So with respect to the D2K-Package, we're quite far and have first versions of the virtual environment, the processes, and the workflow. I hope the rest (adding parameters) is just details ;).

[MarkusKonk]

@IdaMBonnevie I tried to change the year to "2014" and got the following error. So apparently the years 2014-2024 are not available. Shall I just use the listed values? Or is there anything else I need to change? I only changed the year and kept country_code="DE".

[IdaMBonnevie]

Hi Sadra and Markus. Yes, the NUTS years are not consistent. They are supposed to be every 4 years but sometimes the difference is longer. I agree to use the available year options that work. Best regards Ida Den tirs. 18. nov. 2025 kl. 23.57 skrev Sadra Matmir < ***@***.***>:

…

*smatmir* left a comment (AquaINFRA/InteractionPlatform#33) <#33 (comment)> @MarkusKonk <https://github.com/MarkusKonk> You are right. I have tested the base code and received a similar error. Screenshot.2025-11-18.at.23.36.06.png (view on web) <https://github.com/user-attachments/assets/ca060042-939f-43c0-905b-d9446beee155> They merged the annually collected data into the seven-year pack mentioned in error(2003-2024) in Eurostat: Screenshot.2025-11-18.at.23.45.20.png (view on web) <https://github.com/user-attachments/assets/d8703704-5b14-429a-bf9a-c3c394c6fd55> Let's use the seven options highlighted in error between 2003 to 2024. I dont think we have any other option now. I have tested the base code with 2003 and 2024, and it generates the output in the final visualization. @IdaMBonnevie <https://github.com/IdaMBonnevie>, what do you think? — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACI2Y5QY57W3GOJ43CL2NJ335OP7BAVCNFSM6AAAAACLUYE6POVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKNBZG43TGMZWG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[smatmir]

@MarkusKonk , the 'Year' parameter you mentioned is associated with NUTS-3 administrative boundaries (geometries), which is only available every three years.

In Ida Bonnevie's original code, after selecting NUT3 year, she used a filter to select the population data for 2018 only, but in the function that we developed, we consider the latest possible data for the chosen NUT3 year.

It is possible to have two parameters for the year: one for the NUT3 level (with seven different options available between 2003 and 2024) and one for the filtering part, with three or four year options within each NUT3 class.

However, let's not overcomplicate things and just use one parameter for the seven NUT3 classes (between 2003 and 2024) for the Oslo workshop.

Best regards,
sadra

[MarkusKonk]

@IdaMBonnevie Alright, with these inconsistencies it makes even more sense to use a dropdown with predefined values.

@smatmir It's not a problem to add another parameter. So I will use the seven possible year values listed above for param_year here. I just didn't get the "filtering part, with three or four year options within each NUT3 class". I would need some instructions which parameter to change with which values.
But before I do that, we need to agree if this should be done. Any thoughts @IdaMBonnevie?

[smatmir]

@MarkusKonk As mentioned in my previous comment, the parameter you defined focuses on the NUTS-3 geometry year, for which we only have seven available classes between 2003 and 2024.

In @IdaMBonnevie base code, she simply selected the NUTS-3 geometry for 2016 and filtered the population data to 2018 to match the CORINE 2018 dataset, which was one of the workflow inputs (see attached screenshot).:

In the Docker version of the combine_eurostat_data.R function, we fixed the NUTS-3 geometry year to 2021 and used the latest available population data for subsequent analysis in the workflow:

The range I suggested earlier (2014–2024) in our online meeting, refers to the valid range for filtering the population data. This range is to some extend compatible with the CORINE dataset year (2018) and is also the supported year range for the demo_r_pjangrp3 table, which only covers 2014–2024.

My initial suggestion for two parameters in my last comment here, was:
1. One parameter for selecting the NUTS-3 geometry year (2013, 2016, 2021, or 2024).
2. A second parameter for selecting the population table year (2015–2024), depending on the choice of NUTS year.

The relationship between these two parameters (NUTS year and population year) is illustrated in the flow chart you shared:

We then developed a second comprehensive idea today and implemented it:

We replaced "demo_r_pjangrp3" with the "demo_r_gind3" online dataset. This dataset supports population data from 2003 to 2024. In the updated combine_eurostat_data.R function, we added two new parameters:

•	Parameter 1: select from the 7 available NUTS-3 geometry classes,
•	Parameter 2: select from the compatible population years (3–4 options per NUTS class), based on the choice in Parameter 1.

To make the tool easier to use, we also made it interactive: the script now prompts the user to choose a NUTS version and then displays the valid population years for that geometry.

There is only one remaining issue: we still receive warnings and observe an increased number of outliers, particularly when using older population data, such as 2004. These warnings occur because of temporal mismatches— for example, combining 2004 population data with the CORINE 2018 land-cover map results in a 14-year gap. Such mismatches lead to large errors (over 100%) in the dasymetric estimates, which trigger the visualization warnings.

We can address this later by adding additional CORINE datasets for other available years (e.g., 2006, 2012, and 2018) to reduce the temporal mismatch and, consequently, the error rate and number of outliers.

Please find attached the new "combine_eurostat_data.R". It is for idea 2, but can be easily modified for idea 1 as well.
I hope it is useful. if both of you agree with this idea, @MarkusKonk can update the combine_eurostat_data.R in his fork version of the Docker based on the new attached file.

Sadra_New 2 parameters_combine_eurostat_data.R.zip

Best
Sadra

[MarkusKonk]

@smatmir Sorry, I don't get anything and don't have the time now to study this. I also don't know what kind of interaction you added to the function combin_eurostat_data but this is not going to work as a process. And please don't send around code files but use GitHub.

[MarkusKonk]

@IdaMBonnevie I need some input from you regarding the titles and descriptions for the functions and input parameters.
You can find a nice example here: https://github.com/AstraLabuce/aquainfra-usecase-Daugava/blob/containerize/src/ogc_api_processes/points_att_polygon.json

You would need to change the following eight .json files in this folder: https://github.com/MarkusKonk/aquainfra-usecase-elbe/tree/main/src/ogc. Have you received an invitation to become a contributor in my repo? Then you should be able to change the files either in the browser on GitHub directly or after cloning it locally. If you manage to make these changes today, we could send the functions to Galaxy already tomorrow.

[IdaMBonnevie]

Hello Sadra and Markus.

Markus, I'll take a look at the JSON files, hopefully today, I'll try to get it done. Otherwise, it will be tonight the latest. If tonight, will it still be in time for sending to Galaxy tomorrow?

Sadra, nice further developments.

• it seems like a really good idea to make the population year dependent on the NUTS year choice. If you freely could select the NUTS year and population year separately, you can introduce unnecessary errors. It makes sense that if you select e.g. NUTS 2016, you should only be able to select population year 2015; 2016; 2017; 2018; 2019; 2020 - or it would get to close to another NUTS (NUTS 2013 and 2021 respectively). From the user perspective, however, it would make sense to select the population year first though, and then choose the NUTS to focus on, so if you select population year 2019 you could choose between NUTS 2016 and NUTS 2021.
• Nice using a different population table, if it enables annual populations but for a longer time period.
• Regarding the interactive prompting, if this does not work with processes, as Markus states, we could simply leave this out?

[MarkusKonk]

Let's say, if you can make it until tomorrow noon, I will be able to send it to galaxy tomorrow afternoon. If you cannot make, we will need to wait for another week (which is not dramatic but next week is the last chance before Oslo).