Working on the next release

[friendly-bits]

Update: ipv6 support implementation is now complete.

Original post:

I'm working on implementing support for ipv6 and nftables.
What's done by now is re-write of the *fetch script. I have re-written it twice, in fact.

The first time I implemented support for ipv6 and direct fetching from all regional registries, and discarded the code that was utilizing the RIPE API. However, having done that, I discovered that the lists published by the registries are not as straightforward as I thought. There are cases where a subnet that should belong to one registry actually is in the list of another one. So the heuristic of checking which registry covers which region and then only fetching the ip lists from a certain registry based on that - simply doesn't work. I mean, it produces a list that has the large majority of relevant subnets in it, but not all. In addition, some seemingly real subnets are not included in any list of any registry. I don't have an explanation for that but it's a fact of life.

So eventually I decided to do the opposite: use RIPE API and discard the (really much code) that was dealing with fetching and parsing lists directly from the registries servers. Lists fetched this way don't have the flaws described above. One nice feature that got carried over from the previous implementation is checking for updates directly on the server before downloading the actual files. So if there are no updates for a certain region, no files download is performed (except the listing on the server). This is not possible with RIPE API (or at least I don't know how to do that with RIPE API), and I had the code for this already.

I also implemented a custom solution for parsing the json lists fetched through the RIPE API, so now parsing is about 3x faster on a slow CPU (like the one in my router). That won't be much noticeable on an x86 CPU but on embedded, it's a difference of 4.7 seconds for parsing with jq to 1.4 seconds with the new implementation, for a very big list like the US one. If you are running a slow and quite old embedded system and you have say 5 big countries on your whitelist and you use both ipv4 and ipv6 geoblocking, that saves you about 20-30 seconds when fetching and parsing the downloaded lists. Probably not an important gain when running from a cron job, but a noticeable one during the installation when you are waiting to see the results. Plus it removes the jq dependency. Although full-featured GNU sed is possibly required - I'll have to do more testing to see if that works on non-GNU sed or the trimmed-down sed included with some embedded Linux distributions. But the direct-from-server implementation required that as well.
Edit: converted sed implementation to a supposedly portable format so hopefully, this will work with any sed.

So now the *fetch script is basically production-ready but I also want to implement support for fetching lists from ipdeny. The benefits are: 1) it'll be moderately faster because RIPE API has an initial delay when requesting a list from it while ipdeny doesn't, 2) no parsing required, 3) they provide "aggregated lists" which pose supposedly less load on the CPU when processing incoming connection requests. The disadvantages are: 1) it's a 3rd party which may or may not stay online in the long term, and may or may not move to a paid model, 2) no guarantees are provided that it'll be online continuously, 3) the 3rd party has no obligation to provide correct lists, and 4) it is uncertain whether they collect your data when you are fetching the files from them (for example your ip) and then selling that. So considering all that, I think it's a nice fallback feature for people who may experience difficulties with fetching from RIPE, or don't care about privacy and long-term reliability.

Update: implemented support for ipdeny.

The work left to do with regards to ipv6 is adapt the rest of the scripts, which should not be as labor-intensive. As to nftables support, I have not started implementing it yet but I expect that it'll be mostly relevant to the *apply script which will probably need significant modifications.

I'm not pushing the updates to github yet since it's all still a work-in-progress, and the current *fetch script implementation is not yet compatible with the rest of the suite. Keep posted, likely in a week or 2 there should be a new release.