Define file and folder structure

[JamesSample]

We need to agree on a standard file and folder structure for SeaBee data. This needs to be simple enough to be human-readable (e.g. via the MinIO web interface), but structured enough for processing to be automated. Some considerations:

• We should avoid ID codes or unique identifiers in file paths, as these are not human-readable

• The system should allow for sensible groupings of data like, for example, the year => mission => flight hierarchy that is currently used. This will make it easier for people to find what they need

• We need to allow for a range of data types and sources (aerial drones, otters, RGB, MS, HS etc.)

The system used by Sindre for seabirds is pretty straightforward and covers everything we need at present (I think). The system originally proposed by NIVA seems unnecessarily complicated.

As a starting point, how about something like this for the data from each flight:

region_area_yyyymmdd/
├─ annotation/
├─ dem/
├─ gcp/
├─ ground_truth/
├─ orthomosaic/
├─ other/
├─ raw_images/
config.json

Where:

• region is coarse geographic descriptor (e.g. fylke) and area is more specific (e.g. an island, town or building name).
• annotation (optional). Contains any relevant annotation not already in the database, such as geopackages etc.
• dem (optional). Contains elevation datasets generated during orthorectification (DSMs and DTMs etc.).
• gcp (optional). Ground control points in a standard text format. Note: The format Sindre uses for ODM is slightly different to that used by e.g. Spectrofly for Pix4D. Assuming both are supported by both applications, it would be good to pick one as standard, as then we can easily use ODM to reprocess data from Pix4D etc.
• ground_truth (optional). Ground truth data.
• orthomosaic (optional). Georeferenced mosaic images created by ODM or Pix4D. Ideally a single, multi-band GeoTiff
• other (optional). Anything not included in the other folders (reports, logs etc.)
• raw_images (required). Images from a single flight (i.e. images that can be orthorectified to produce a single mosaic). This could also include raw images from e.g. the otter, as long as they are to be stitched together
• config.json (required?). I'm still not certain whether this is useful. However, this file could include settings and metadata to control subsequent processing. For example, it could include the Dronelogbook ID (in which case, additional metadata could be extracted via the API during processing), or it might include publication options (e.g. to control whether a dataset is made publicly available on GeoNode). Thoughts?

The general idea is that researchers could organise their data on MinIO more-or-less as they wish, as long as each flight is organised as illustrated above. For example, on MinIO we might have:

niva/
├─ 2022/
│  ├─ runde/
│  │  ├─ more-og-romsdal_remoy_20220831/
│  │  │  ├─ annotation/
│  │  │  ├─ dem/
│  │  │  ├─ gcp/
│  │  │  ├─ ground_truth/
│  │  │  ├─ orthomosaic/
│  │  │  ├─ other/
│  │  │  ├─ raw_images/
│  │  │  ├─ config.json
nina/
├─ 2022/
│  ├─ rogaland_risavika_20220627/
│  │  ├─ dem/
│  │  ├─ gcp/
│  │  ├─ orthomosaic/
│  │  ├─ other/
│  │  ├─ raw_images/
│  │  ├─ config.json

This is just meant as a starting point for discussion. What do you think, @knl88 @deviirmr @awigeon ?

[deviirmr]

I like the above folder structure and agree that we need to follow a simple human-readable name conventional. There are a few additions to the current structure based on the GUI interface we discuss in the past

niva/
├─ 2022/
│  ├─ runde/
│  │  ├─ more-og-romsdal_remoy_20220831/
│  │  │  ├─ annotation/
│  │  │  ├─ dem/
│  │  │  ├─ gcp/
│  │  │  ├─ gtp/
│  │  │  ├─ ground_truth/
│  │  │  ├─ orthomosaic/
│  │  │  ├─ raw_images/
│  │  │  ├─ drone_path/
│  │  │  ├─ other/
│  │  │  ├─ config.json

I have added gtp (Ground Truth Point) and drone_path (maybe geojson or kml) but we can rethink whether we needed those inputs or not.

In the config.json file, we can also include pre-define information such as

• weather parameters
• drone sensor detail (such as type of drone and sensor)
• data contact point for the uploaded data (piolet/user name etc).
• may be some configuration details for the machine learning

[JamesSample]

Thanks @deviirmr .

ground_truth is already included in my proposed structure. Or did you intend something more specific for the gtp folder you added?

[knl88]

Think the below is fine, with no space,spesial characters etc. Following the logic, should it be ground-truth and raw-images, if underscore is used as seperator?

region_area_yyyymmdd/
├─ annotation/
├─ dem/
├─ gcp/
├─ ground_truth/
├─ orthomosaic/
├─ other/
├─ raw_images/
config.json

hmm, guess we can see what we need to put into config.json:) .. and would be cool with a small json schema! This could perhaps be a discussion on its own, but agree it could be some processing and/or publications options, for publishing it would also be nice to have a title and abstract then I think we could also generate the iso record fairly quickly. The later is slightly of topic if we just think the config.json as configuration for the pipeline, although one of the processing elements would need this as input also.

[JamesSample]

Good points!

I agree regarding ground-truth and raw-images.

[smolvarsmyr]

I feel a bit stuck in the odm way of doing it, but I will try to widen my view slightly to include niva data better.

• We need to add somewhere for 3d models/texturing. Havent really used it yet, but if someone figure out a nice way to use them in the field/on a touchscreen it could be extremely useful.
• For me it seems like the gcp files for pix4d and odm are different, and i dont think they accept each others files. They are easy to translate though.
• report.pdf is very important to quickly see the results. I use stdout alot as well. Could either be report/ or other/
• other/ feels like a place that could be very messy, but its outside any pipeline i guess, so that is okay for me
• nina will probably not use annotation or ground-truth
• Word changes: orthomosaic or orthophoto? raw-images or images?

region_area_yyyymmdd/
├─ annotation/
├─ dem/
├─ gcp/
│  ├─ gcp_list-ODM.txt
│  ├─ gcp_list-Pix4D.txt
├─ ground-truth/
├─ orthomosaic/
├─ other/
├─ raw-images/
├─ report/
│  ├─ report.pdf
│  ├─ stdout.txt
├─ texturing/
config.json

A native odm installation or docker version uses

├─ images/
gcp_list.txt

as its entire input. Do we want to follow that? I think I have concluded we dont have to, and to keep gcp in a folder might be more clean.

[deviirmr]

Thanks @deviirmr .

ground_truth is already included in my proposed structure. Or did you intend something more specific for the gtp folder you added?

Sorry, I miss the entries while reading the structure, you are right you already included the ground_truth (gtp)

[deviirmr]

niva/
├─ 2022/
│  ├─ runde/
│  │  ├─ more-og-romsdal_remoy_20220831/
│  │  │  ├─ annotation/
│  │  │  ├─ dem/
│  │  │  ├─ gcp/
│  │  │  ├─ ground_truth/
│  │  │  ├─ orthomosaic/
│  │  │  ├─ other/
│  │  │  ├─ raw_images/
│  │  │  ├─ config.json

@JamesSample : The year in this structure is the data upload year or its flight mission year?

[JamesSample]

Looks good @awigeon !

(Sorry for the slow reply).

I'm happy with all your suggested changes i.e.

region_area_yyyymmdd/
├─ annotation/
├─ dem/
├─ gcp/
│  ├─ gcp_list-ODM.txt
│  ├─ gcp_list-Pix4D.txt
├─ ground-truth/
├─ orthophoto/
├─ other/
├─ images/
├─ report/
│  ├─ report.pdf
│  ├─ stdout.txt
├─ texturing/
config.json

However, I've just had a quick look at reorganising some of the old NIVA data and it's going to be fiddly. In particular:

• There are sometimes several flights at the same location on the same date. These are currently distinguished by time, which implies e.g. region_area_yyyymmdd-HHMM in the top-level folder name.

• There are even some flights with the same location, date and time (presumably MS and RGB cameras on a single drone?). So then we need e.g. region_area_spec_yyyymmdd-HHMM.

What do you think? We could give it a go, but it feels like it's getting a bit messy. Or we could just use the structure above and then organise as e.g.

niva/
├─ year/
│  ├─ mission/
│  │  ├─ rgb/
│  │  │  ├─ region_area_yyyymmdd/
│  │  ├─ ms/
│  │  │  ├─ region_area_yyyymmdd/

Then there are cases where data from multiple flights has been combined into a single mosaic... argh!

@deviirmr

The year in this structure is the data upload year or its flight mission year?

It's the flight mission year.

[smolvarsmyr]

I also sometimes combine images from multiple flights into the same mosaic (and it also varies a bit how the drone handle battery swaps, sometimes it makes a new flight log). Usually that is simply done by just adding all photos from multiple flights into the images folder. But I see the problem when you want to connect to dronelogbook. This is also kind of a reason why i havent gone down that route.

With regards to multiple missions on the same site the same time: I have done that by region_area-1_yyyymmdd and region_area-2_yyyymmdd, where 1 and 2 could be whatever. Sometimes i have also rerun mosaics with new settings or gcps and just made a new "mission" that way. Could do the same with RGB vs MS.

[HegeGundersen]

Ah, this is so needed! Thank you @JamesSample for the initiative and all for good suggestions. My only comment at this point is that I think its very useful to have Project or Mission (e.g. MASSIMAL or KELPMAP) high up in the hierarchy (maybe instead of Region). In many cases the Mission is actually the Region, so then it will be the same.

[smolvarsmyr]

@HegeGundersen In secret I do this already, ie. writing project or some other grouping instead of an actual region. I would say region could be used more to group the flight in the folder, in however way you like. Dont really see a problem with that. Georeferencing comes from the files anyway.

@JamesSample I usually have solved this by simply

niva/
├─ year/
│  ├─  grouping_areaRGB_yyyymmdd/
│  ├─  grouping_areaMS_yyyymmdd/
│  ├─  grouping_areaRGB1_yyyymmdd/
│  ├─  grouping_areaRGB2_yyyymmdd/
│  ├─  grouping_areaMS1_yyyymmdd/
│  ├─  grouping_areaMS2_yyyymmdd/

I think that is a more clean folder structure, and think is is kind of flexible enough to cover all cases. It is however slightly hard to computer read. Maybe adding a _ or - somewhere

[HegeGundersen]

One more thing: often NIVA and SpectroFly joins the same campaigns. It will then be useful to have their datasets grouped and not in completely different institution folders. But this will maybe f... up the whole structure...?

[JamesSample]

@HegeGundersen If we can agree on a grouping/naming convention for the data from each unique flight-camera combination (something like the structure proposed by Sindre, above), I don't think this is a big problem in terms of reading data automatically - users can group the separate flight folders however they want, including e.g. having a single project folder with data from both NIVA and Spectrofly.

The code on the platform can just search for folders containing a file named e.g. seabee-config.json. Any folder containing a file like this would be identified as a "flight folder" and assumed to follow the subfolder structure outlined above, no matter how deeply nested it is in the "parent" folder structure.

Mixing data from different organisations does make it harder in terms of browsing folders manually, though. The original suggestion was to make separate "buckets" for each organisation (niva, nina, ntnu, spectrofly etc.) and then each would do something like what Sindre proposes in his post above. If we want the NIVA and Spectrofly data together, then for manual browsing we might need to add the organisation to the name of the flight folders e.g.

grouping_area_org_spec_num_yyyymmdd/

We can try it like this, but it gets increasingly fiddly and fragile. Ideally, I want to avoid having to extend the folder naming conventions to cover all the edge cases, because then I think we'll end up back with something like the current NIVA structure, which is hard to work with.

[smolvarsmyr]

I would vote strongly to keep it simple, and avoiding org as a required field. Does it really matter who it was flown by? What about having only three required fields, seperated by _, and then adding all the extra info after -

grouping_area-orgspecnum_yyyymmdd
Runde_Tarevågen-NIVARGB1_20220818

Or actually, if it just scans for folders with config file, the naming scheme of there parent folders probably doesnt matter at all.

[JamesSample]

I agree about keeping it simple.

In terms of automatic processing, none of this matters - the easiest option would probably be to have a random database ID for the folder name and all the metadata in config.json.

However, I think we also need something that is human-readable, because users still want to browse the data via e.g. the MinIO UI. In this context, it is important that people can identify which datasets are from NIVA and which from Spectrofly etc., without having to open config.json or explore other file metadata. Ideally, it should be obvious from the folder names.

I like your suggestion above. Or maybe just have the first three elements mandatory, then optional info afterwards

grouping_area_yyyymmdd_org-spec-num

?