Importing or loading GSAK databases?

[CheminerWill]

Will we be able to “import” or otherwise use existing personal GSAK databases? Most of my GSAK databases are not an issue to recreate. But a few, especially my “local” (within 150 miles of home), data base would be impossible to recreate with all its existing data. It has 15,000+ geocaches many of which have been updating logs, etc. for 10 or 20 years and some caches have hundreds of logs. That database regularly loads about 20 pocket queries, using the “download pocket queries” tool in geocaching.com access in GSAK. Then I filter for what I want and export those to my GPSMap 66sr using the garminexport.gsk macro. When I do begin using OpenSAK, it would be great to be able to use this existing home area database.

EDIT: I am exporting my largest database as a gpx file, then will import it into OpenSAK and see if that gets me all my information from my GSAK database. It is taking a long time as the db is so big, but I will report the results here when it completes.

[CheminerWill]

That pretty much worked. There are a few quirks I will try to figure out on my own first to see what might be going on. For one example even though a cache might correctly show in OpenSAK count of logs, only 20 of those are able to be viewed. Even if the OpenSAK database says there are 48 logs, only 20 show. All 48 are shown in GSAK. I can't find a setting to change that.

[Fabio-A-Sa]

Thanks for sharing it. Currently we do not expect any changes between GSAK and OpenSAK format, but it may have an issue on logs import, since as you described it only shows (or even imports!) just 20 for each geocache. I will create an issue for further investigation.

Issue: #266

[AgreeDK]

it was a display cap in our GUI, so it will be corrected in our next beta release.

[CheminerWill]

OK. Yes eventually it should be able to import all the logs as having them all is one of the reasons GSAK users keep their databases updated for so many years. Strange that OpenGSK shows the correct number of logs in the count even though only the first 20 are there. The program seems to pick up on how many there are to import, puts that number into the program, but then only imports the first 20. I scrolled through several with high log counts, some in the hundreds. All stated that there were the correct number of logs, but when you look at the scrollable list of logs, there are always only 20. Thanks for creating a ticket/issue for future resolution. Edit: I checked my GSAK export settings to make sure I did not have it set incorrectly there. It is set as it should be with "no limit" to the number of logs to include. Then just be sure the issue was not with the exported gpx file, I imported it back into a test database in GSAK and all the logs show in the list as they should. So indeed the bug must be in the OpenGSK import process I guess. Thanks!

[stoppie777]

Hi, in your GSAK import test, did any Userdata fields get copied across? I cannot get GSAK wpt:extensions in the gpx file to be recognised by openSAK import. I understand that the Custom fields are ignored but expected the Userdata to be imported.

[CheminerWill]

@stoppie777 I am not 100% sure what "Userdata" fields are? If you can specify for me what to look for I will check. If "found by me" is userdata, then yes, I have those caches noted. But that brings up another thing about how those found by me caches are shown. They are green. Maybe it is just me, but I am used to seeing those noted in yellow. Green has always meant something else.

EDIT: Just noticed that even though my found caches are shown in OpenSAK, my owned caches are not. So not all data like that is being brought across. Hope that helps answer you question. Just let me know if there are any other fields you would like me to check. The database I brought across has over 15,000 caches so lots of data.

[stoppie777]

There are several fields that appear in both GSAK and openSAK that do not appear to be imported to openSAK from a GSAK gpx extract. I have done some testing and looked at the code and it appears some fields have not yet been included in the Importer. To answer your question there are 4 fields in GSAK that you can add any relevant data you wish, very useful in special filters. I believe the 'Found By me' is derived from the logs. As for the colours, I haven't really noticed in openSAK, as it is not a direct clone of GSAK, so I expect a lot of cosmetic changes.

I have added screenshots below showing the UserData fields.

from my test data I noted the following which I hope the openSAK team ( @Fabio-A-Sa ) will see in this discussion as it falls under the same heading, 'Importing or loading GSAK databases'.

GSAK WptExtensions not imported but exist in OpenSAK

gsak:UserFlagtrue</gsak:UserFlag>
gsak:CountySomeCounty</gsak:County>
gsak:IsPremiumtrue</gsak:IsPremium>
gsak:FavPoints144</gsak:FavPoints> but would require Geocaching.com API if not from GSAK generated GPX
gsak:UserSort0</gsak:UserSort>
gsak:UserDataUD1 test</gsak:UserData>
gsak:User2UD2 test</gsak:User2>
gsak:User3UD3 test</gsak:User3>
gsak:User4UD4 test</gsak:User4>

Already implemented:
src/opensak/importer/init.py

ftf_el = gsak_ext.find(f"{{{gsak_uri}}}FirstToFind")
before_lat_el = gsak_ext.find(f"{{{gsak_uri}}}LatBeforeCorrect")
before_lon_el = gsak_ext.find(f"{{{gsak_uri}}}LonBeforeCorrect")

gsak:FirstToFindtrue</gsak:FirstToFind>
gsak:LatBeforeCorrect54.169017</gsak:LatBeforeCorrect>
gsak:LonBeforeCorrect-4.5259</gsak:LonBeforeCorrect>

[CheminerWill]

Got it. Seems like you have it figured. I don't use those 4 user data fields so can't say if they would have come over for me. I agree with the cosmetic differences but for things like found, owned, etc. it may be better to match up to the geocaching.com website, their app, and to GSAK.

[AgreeDK]

I will raise an issue to look at the import/export of GSAK WptExtensions so it can be handled correctly.

[AgreeDK]

Import side of this is now implemented and tested.
_parse_wpt() reads gsak:UserFlag, gsak:IsPremium, gsak:UserSort, gsak:UserData/User2/User3/User4, and gsak:FavPoints from GSAK's block, and _upsert_cache() writes them onto the existing Cache columns (added back in #33). Values are only overwritten when GSAK actually supplies them, so a plain Geocaching.com PQ re-import won't wipe data that came from a previous GSAK-sourced import.
gsak:County wasn't added separately — county is already populated from the standard Groundspeak gs:county field, so that one's covered.
Tested against a real GSAK export (Windows, GSAK test DB with UserFlag/Premium/Sort/UserData1-4/FavPoints filled in) — all fields came through correctly in OpenSAK's column view.
This will be included in the next beta release (v1.14.0-beta.6).
@Fabio-A-Sa — export (writing these fields back out to a GSAK-compatible GPX) is being deferred to a separate issue. OpenSAK doesn't currently have a GSAK-format export path at all, so that's a bigger piece of work on its own rather than a quick add-on here.