New dataset links

[mateoclarke]

@haileypate Do y'all have a working list of all the new datasets?

It's be great if they were available by department so that this progress report could lead you into discovering the newly published data by dept. If the list were published through Socrata, we could grab it through another API call. 😉

[haileypate]

Great suggestion. My working list has "all" of the datasets, and I could filter that and push it back up to Socrata. The question is... how shall we define "new" dataset?

This is an interesting question. The data we have available gives us "creation date" and "last updated" for each Socrata resource ID. The thing is... when a dataset owner adds a new dataset by going to an existing resource, wiping out the columns and/or metadata, and replaces it with the new data... this won't get captured by filtering on creation date. In some cases where departments have been making good faith efforts to update existing datasets, we've been giving them credit during this first sprint.

We're going to have to do some outreach and guidelines to set a process for how resources as created, updated, and deprecated so we can get a better handle on what everyone's really doing with data. More importantly, though, this will be an important part of delivering a good user experience for folks making API calls.

So in a nutshell, I don't have the list today, but it's something we're working on and would definitely share when it materializes. I can put up the entire list in the meantime if you like. Stuart from Socrata told me last week that he'd turn on a feature that automatically creates this dataset for us. I'll be curious to see how that compares with what my spider captures ;)

[haileypate]

I know I'm a bit behind on this... but I promise it's in my queue.