feat: implement IPtoASN -> mmdb (GeoLite2-ASN / GeoLite2-Country) pipeline - #1
Merged
zxdev merged 4 commits intoJul 22, 2026
Merged
Conversation
…ribution required
Address review nits on the IPtoASN -> mmdb pipeline: - iptoasn: gofmt the package doc comment (was not gofmt-clean). - mmdb-build-countries: pickContinent is now deterministic (sorted) and transcontinental countries are pinned via continentOverride, so regenerating countries.json is reproducible instead of depending on map iteration order. Drop a duplicate, unreachable err check in main(). - mmdb-write: extract versionString() so it is testable; document that registered_country mirrors country and that country pseudo-codes (EU/AP/ZZ/Unknown/empty) are skipped. - tests: replace the tautological TestVersionFlag with TestVersionString; add TestCountryPseudoCodesSkipped and TestCountryAmericanSamoa (the "AS" ISO code vs "AS" ASN-prefix collision), and TestParseSpecCountryFallback in the core parser covering the same collision. No change to the shipped countries.json; the override values match it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new tool,
mmdb-write, for compiling CIDR specs into MaxMind DB files, and extends theiptoasntool to support both ASN and country output formats, similar to themm-dbiptool. It also adds a build/install script for automated MMDB generation (both GeoLite2-ASN and GeoLite2-Country format) and updates documentation for all command-line tools.The new
mmdb-writetool expects, in standard input (or an input file using the-iflag) CIDR-format input data (<cidr> <ASN>or<cidr> <country>), and writes a MaxMind-compatible database from the given IP ranges. In effect,mmdb-writecan be used in Unix-style pipelines to convert any CIDR data to standardizedMMDBbinary format.For country data, we need to enrich the input CIDR dataset with a countries nomenclature. This nomenclature is stored in
cmd/mmdb-write/data/countries.jsonand is generated by themmdb-build-countriesutility. Originally, I used the geonames.org dataset, which is released under a CC attribution-required license. The original commit still contains this implementation: it requires attribution; the latest commit replaces this with a pure-Wikidata source. Two Wikidata SPARQL queries are used to query continents and countries, translated into the same languages MaxMind's native DB uses.The resulting databases are fully compatible with any library / program using
GeoLite2-ASNorGeoLite2-Countrydatabases. The format itself is not bound by licensing terms - the reader libraries are available under ISC (no warranty given, but no attribution equired) - the country data is available under CC0 (no attribution required), if generated with IPtoASN data, the ASN IP range mappings are public domain - the resulting database (IPtoASN -> MMDB) can be used commercially, without attribution required.The Go library remains standard library only. Only the
mmdb-writecmd requires third-party MMDB libraries - this utility has a differentgo.mod. The maingo.modremains empty.For example, if I want to compile the latest IPtoASN ASN data to a new
iptoasn-asn.mmdbfile:To compile the latest IPtoASN country data to a new
iptoasn-country.mmdbfile (uses country field of ASN):Similarly, to compile DB-IP to
dbip-country.mmdb:Feel free to inspect the MMDB response to compare to a MaxMind DB: