Skip to content

Modernize, minimize, and cleanup containers#129

Merged
jcpunk merged 36 commits intodevelopfrom
modernize
Feb 2, 2026
Merged

Modernize, minimize, and cleanup containers#129
jcpunk merged 36 commits intodevelopfrom
modernize

Conversation

@jcpunk
Copy link
Contributor

@jcpunk jcpunk commented Jan 2, 2026

There is a lot going on here. An AI summary exists further down the comment history.

I recommend reviewing the commits one at a time rather than the full diff as a blob.

In theory each commit is a rational step with clear self contained logic.

Copilot and coderabbit identified bits of code that probably never worked. These are probably bits that could be dropped, but that would require an expert to review.

The changes to the application specific code should probably be reviewed.
With NP04 offline right now I couldn't check against the current runtime for some things.

To be clear, I really only care about the containers as containers in kubernetes...

I've tried to set the docker build step for the microservices container to run after the microservices_dependencies container. Those workflows wont really work until this PR is merged. After that everything should be fine...

Copy link
Member

@eflumerf eflumerf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Things look reasonable to me, however I do not have any kind of testing environment so I can't speak too much about functionality.

@jcpunk jcpunk force-pushed the modernize branch 12 times, most recently from 72b65f9 to c8d8f52 Compare January 2, 2026 22:19
@jcpunk jcpunk marked this pull request as draft January 3, 2026 05:12
@jcpunk jcpunk force-pushed the modernize branch 12 times, most recently from 284efa9 to 1258c6a Compare January 3, 2026 20:08
jcpunk and others added 24 commits January 23, 2026 10:48
The get_login method now accepts an optional user parameter, but when user is provided and no matching authentication is found, the method returns None implicitly. This could lead to NoneType errors in calling code. Consider raising an exception when the requested login is not found, similar to how the error is logged but execution continues.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The function returns False here, but earlier in the retry loop at lines 140 and 196, successful operations return True. This creates an inconsistency: the function may return True, False, or raise an exception. Consider either always raising an exception on failure (removing the return False) or documenting the mixed return behavior clearly.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The get_login method has inconsistent error handling. When a service is not found and no user is specified, the method logs an error but doesn't raise an exception (line 198), potentially returning None. This could lead to AttributeError later when the caller tries to access attributes on the returned value. Add a raise statement after line 198 to maintain consistency with the user-specified case.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The ensure_required_variables helper logs the value of every environment variable whose name does not contain one of the hard‑coded sensitive keywords (USERNAME, PASSWORD, DATABASE_URI), which will expose any other secrets (e.g. API_TOKEN, JWT_SECRET) to container logs. An attacker or unprivileged operator with access to logs could recover these credentials and use them to access downstream services or databases. To avoid leaking secrets, stop printing environment variable values entirely (or maintain a strict allowlist of non‑sensitive variables) and only report whether each variable is defined or missing.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Use shutil.which rather than custom function
@jcfreeman2
Copy link
Contributor

Two changes below which I suggest; first an overview:

I'll add the caveat that until this review I hadn't looked at any microservices code in over two years, and I don't know how to run it in a live situation the way, say, Marco does. Having said that, my take on what I see is the following:

  • It's nice to see additional error checking in various locations where it was simply missing
  • For security purposes, not automatically printing out all environment variables for a given microservice strikes me as a good idea
  • Similarly, allowing the microservices to run as not-root is also a good idea; I recall not being able to get that to work a few years back

Now for things it would be nice to update:

  • In Dockerfile.dependencies, the versions of the packages whose files are pulled are unchanged from their original 2023-era values. At least in the case of kafkaopmon's python/kafkaopmon/OpMonSubscriber.py, the code has since been changed, so it would be nice to bring the versions up to date.
  • Documentation (basic suggestion): Some of the obsolete microservices have been dropped, but the docs/README.md doesn't reflect this. Also some of the nomenclature in the documentation needs to be brought up-to-date - e.g., microservices_dependencies gets renamed in this PR, so the documentation should reflect it.
  • Documentation (deluxe suggestion): For a normal (as opposed to power) user of microservices, if I actually want to run a full blown microservice (as opposed to watch it error out on daq.fnal.gov), what do I need to do? Where do I run it? What credentials do I need? Etc.

@jcpunk
Copy link
Contributor Author

jcpunk commented Jan 26, 2026

In terms of python package version updates, I fear that is "not me"... I'd love to "just use" protobuf 6 throughout the stack.

I've updated "documentation basic" I think, for advanced usage I fear that is outside my use of these containers.

Copy link
Contributor

@jcfreeman2 jcfreeman2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

@jcpunk
Copy link
Contributor Author

jcpunk commented Jan 26, 2026

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants