Registry and Plugin (Backend)

[psilabs-dev]

• Issue: Implement dedicated Plugin Repository #1146
• Plugins demo registry repo: https://github.com/psilabs-dev/lrr-plugins-demo

Under testing, lots of code may be subject to change. There are quite a number of avenues where code might break. I don't recommend reviewing this code atm.

This backend-only PR removes all (non-local) plugins from LRR, and implements the registry system and plugin management.

• Multiple registry support
• Management features, such as installing/uninstalling/upgrading plugins, plugin conflict resolution, management APIs.
• Support for installing/uninstalling managed plugins
• Redis is now responsible for plugin package presence management (previously INC's). Which makes uninstall/install easier.
• Configuration-related features, such as metadata plugin priority, enablement, and dropdown hiding. Add metadata plugin priority support.

Obligatory security warning, as downloading untrusted arbitrary files and allowing code execution from the internet requires extra care which was not a consideration when plugins were the responsibility of LRR. The boundary for this PR is that LRR will not audit or review files/registries/licenses for the user, the user must trust registries and audit them personally/take responsibility for any bugs/vulnerabilities within plugins.

Some examples of ways to be compromised, which LRR cannot protect against: registry URL typos/typo squatting, DNS poison/certificate updates, hijacked registries/supply chain attacks

(tbh, with this I'm also just wondering whether plugin installations should require a restart altogether)

(PS: maybe we should support uninstalling sideloaded plugins?)

[psilabs-dev]

Updated this PR so it doesn't touch existing plugins, that's probably a separate concern to decide what to add/remove from LRR.

[psilabs-dev]

An update on duplicate plugins. There are many ways duplicates can happen with the entire plugin system, and the existing dev logic doesn't really handle it besides choosing a survivor and moving on. Now that a subset of plugins have crud, a duplicate does happen, then we might also run into myriad of other unforseen issues.

At the moment, I don't think this PR will guarantee graceful duplicate handling, or handling of issues caused by duplicate plugins.

LRR will perform plugin discovery on startup and report duplicates at warning level, and this PR will ensure that LRR plugin APIs do not introduce duplicates into the system. But past startup, LRR assumes a clean plugin state, and may not handle cases where a duplicate existed or is introduced by the user via other means.

(unless loc budget/complexity isn't a concern)

[psilabs-dev]

Removed channels in favor of semver, all registry plugin versions will be enforced at "x.x.x" style, and LRR will resolve installations by max semver version. Users can opt to install specific versions.

Added stack-based rollback on install failure. If LRR fails to upgrade a plugin, for example, it's going to attempt to restore the previous plugin (which is an existing stance iirc). This is a one-time rollback; if the rollback itself fails, then we're not going to "rollback the rollback", instead return 500 error.

Probably the (remaining) big item is where to place sideloaded plugins, I'll think about that after the local/git registry system is stable.

Commit history is mainly for tracking what's happened. Once the pr matures (after I do another pass), I'll do a rebase and squash.

[psilabs-dev]

The controller layer and OpenAPI contract are fixed, with exception of maybe some cosmetic changes but I won't be touching controller files anymore (unless something's really wrong). So this branch is free to api test.

Next up will be improving model/utils-side of things. A lot of the code is validation and exception handling work, not exactly business logic, but I'll have to see what exactly needs changing instead of just moving things around.

[Guerra24]

I don't see a way to list available plugins by registry, is it part of another pr or I'm just blind?

I have a feeling empty strings will cause issues in the api.

Code looks fine maybe a bit overly verbose at least compared to the rest.

I will do a second pass when I get to test it on Windows.

[psilabs-dev]

I don't see a way to list available plugins by registry, is it part of another pr or I'm just blind?

No this isn't added

[psilabs-dev]

One note: any request/response validation that can (and should) be done by OpenAPI won't be re-done by LRR. So this PR assumes that OpenAPI validation is enabled for any testing.

[psilabs-dev]

Removed the metadata plugin features as I think we already have enough to go over as is

[psilabs-dev]

basic cdn support is added

[psilabs-dev]

Model layer (and above) are now reviewable (assuming the tests pass). Next up will be improvements to the Utils-layer.

[Guerra24]

Looks fine to me, tested it on Windows and it works as expected. Model layer looks cleaner and is easier to follow.

[psilabs-dev]

The Utils layer (Plugin/PluginState/Registry) should be stable now. Going to start dogfooding on this with a gitlab/codeberg account + self hosted gitea.

Unit and integration tests are still AI written, maybe we'll go through another round of mutation testing before I open it up for review.

[psilabs-dev]

squashed, backup at https://github.com/psilabs-dev/LANraragi/pull/new/dev-registry/backend-backup

[psilabs-dev]

Checked that registry on self-hosted gitea can be read by LRR (after some cert/dns shenanigans).