Generate JsonLD datasets from websites

[valentinedwv]

There is a google sheet with earthsurface processes.

Can we use AI to do:

1. generate a JSON-LD webpage description

Spend a little human time, or ask AI if it sees datasets. eg the merit dem site has several.
and generate JSON-LD.

So, need to generate a series/set of prompts to do this.

addendum:
One time I tried this. And Doug had a project that used BAML to manage whatever BAML does.

siteFromPrompt

Experiments

[ywkim312]

@valentinedwv What is the " merit dem site"?

NVM I found it from the google sheet

[ywkim312]

@valentinedwv @ylyangtw I've been working on the JSON-LD generation for the datasets in the Google Sheet. The script is set up, and I've tested fetching webpages (like the MERIT DEM site), so that part works.

To finish, I need an API key for an AI service to generate the JSON-LD. The script supports OpenAI (like GPT-4) or Anthropic (Claude), and we can add others if needed.

A few questions:

• Do you have an API key we can use, or should we set up a new account?
• Any preference on which service (OpenAI, Anthropic, or something else)?
• Any budget or usage limits to keep in mind?

Once we have the key, I can test it on the MERIT DEM site and then run it on all the datasets that need JSON-LD. Let me know what works best for you.

[valentinedwv]

Lets use the National Research Platform LLM
https://nrp.ai/documentation/

[ywkim312]

Lets use the National Research Platform LLM
https://nrp.ai/documentation/

Do you have a api key for this? Or should I create one myself?

[valentinedwv]

Sent you a new one.
You can also create these, yourself

[ywkim312]

Sent you a new one. You can also create these, yourself

I got it. Thank you!

[valentinedwv]

Once wrote a prompt, and actually stored it.

experiments/siteFromPrompt

[valentinedwv]

Think we might need to rethink this, and put a human in the loop with our partners.

You want to collect multiple datasets from merit DEM


On the sidebar under links there are datasets, like MERIT DEM and MERIT Hydro

file naming convention:
The naming format is:
Filename represents the center of the lower left pixel of the data domain; e.g. the file "n30w120_dir.tif" is for the Flow Direction of the domain N30-N35, W120-W115. (more accurately, N29.99958333-N34.99958333,W120.0004167-W115.0004167)

The 5 deg tiles are compressed into 30 degree (lat) by 30 degree (lon) package.
Package name represents the lower left corner of the package domain; e.g. the package "dir_n30w120.tar" contains the files in the domain of N30-N60, W120-W090.


If you go to:
http://hydro.iis.u-tokyo.ac.jp/~yamadai/MERIT_DEM/index.html
There is a product description, and download links for multiple data formats, ESRI, GeoTIFF, and MMR
this should be one datacatalog, with files for different spatial regions. 



if you go to:
http://hydro.iis.u-tokyo.ac.jp/~yamadai/MERIT_Hydro/index.html
There is a product description, and download links for multiple datasets

Flow Direction (Local Drainage Direction)
Hydrologically Adjusted Elevations
Upstream Drainage Area (Flow Accumulation Area)
River Channel Width
HAND: Height Above Nearest Drainage