Add packages inventory collector#15
Merged
Merged
Conversation
sergeyfast
force-pushed
the
feat/packages-collector
branch
from
8fd35ed to
69fe330
Compare
- Add internal/topsrv/packages: Dpkg/Apk/Rpm managers as concrete
structs with methods (no interface — concrete types are clearer
per the project's philosophy). detectManagers() returns dispatch
entries by probing marker files
- Parse dpkg status, apk installed DB, rpm BDB/NDB/SQLite via
anchore/go-rpmdb v0.1.0; share lock-safe read pattern with Trivy
(copy-then-parse for BDB, immutable=1 DSN for SQLite)
- Enrich with apt extended_states (autoInstalled), dnf history.sqlite
(autoInstalled + repoOrigin), /etc/apk/world (autoInstalled);
cover NEVRA, GPG keyid, modularityLabel, vendor, sigDigest,
licenses for Vulners CVE matching on gatesrv
- Add /v1/inventory endpoint with kind discriminator; new
InventoryProvider + InventoryAckReceiver interfaces alongside
QueryMetaProvider; spool to SpoolDir/inventory/<kind>-<ts>.json
- Expose topsrv_packages_{installed,held,scan_duration_seconds,
scan_errors_total,last_scan_timestamp_seconds,
last_push_timestamp_seconds,manager_info}
- Refactor push.go: extract postJSON + spoolFile helpers, replace
deriveMetaEndpoint with generic deriveEndpoint, fold trimSpool
inline so all spool subdirs share the cap
- Wire registerPackages in app.go and [Packages] block in
local.toml.dist; opt-out via Disabled/DisablePush
- Add unit tests for Rpm.parseSrcRpm, Dpkg.parseSource,
Dpkg.parseRFC822, Apk.setChecksum, Rpm.extractKeyID
- Depend on github.com/anchore/go-rpmdb v0.1.0 and
modernc.org/sqlite v1.50.1 (pure-Go, no CGo)
sergeyfast
force-pushed
the
feat/packages-collector
branch
from
69fe330 to
4fd9ad5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
internal/topsrv/packagescollector reads dpkg/rpm/apk databases pure-Go (no CGo, no shell-out) and exposes counts astopsrv_packages_*Prometheus metrics/v1/inventoryendpoint, parallel to existing/v1/metafor postgresinternal/topsrv/push.go: extractpostJSON+spoolFilehelpers, generalize endpoint derivation, fold trim into spool — single source of truth for both metrics and inventory transportDesign
*Dpkg/*Rpm/*Apkwith methods, no interface — per project philosophy)/var/lib/dpkg/status,/var/lib/rpm/*,/lib/apk/db/installed); auto-fallback through rpm sqlite → ndb → bdb?mode=ro&immutable=1(Trivy pattern)kindon/v1/inventory(gatesrv routes by discriminator)SpoolDir/inventory/<kind>-<unixms>.json, trim topushMaxSpoolSize=100github.com/anchore/go-rpmdb v0.1.0+modernc.org/sqlite v1.50.1(pure-Go)Docs
docs/packages-collector-research.md— industry comparison (Syft / Trivy / osquery / Wazuh), why/v1/inventorynot extended/v1/metadocs/packages-collector-implementation.md— full contract, security data model (MUST / SHOULD / NICE per Vulners audit), concurrent-safety analysis, overhead measurements, build-tag plan for slim buildsTest plan
make fmt lint test— 0 issues, all greenmake test-integration— postgres / nginx / angie / smart / botlog integration tests green (no regressions in transport refactor)Rpm.parseSrcRpm,Dpkg.parseSource,Dpkg.parseRFC822,Apk.setChecksum,Rpm.extractKeyID) under//go:build linux, run via OrbStack:9100, payload accepted by local gatesrv →/v1/inventoryreturns 204Out of scope (Phase 4 / future PRs)
kind="repos"snapshot (apt sources.list, yum.repos.d, apk repositories + GPG keys)kind="packageHistory"snapshot (apt history.log, dnf trans, apk.log)no_rpm_sqlitefor slim binary (drops sqlite runtime, -6 MB / -14 MB RSS, loses RHEL 9+ sqlite rpmdb support)testdata/packages/)repoOriginreconstruction — analysis recorded in implementation doc; not done by Syft/Trivy/osquery/Wazuh for cost reasons,vendor+kind="repos"cover the security signaltopsrv_packages_upgradablemetric (depends onCheckUpgrades=truereading local apt/dnf cache — field reserved but not wired)