Dependency Updates, Speed Improvements, Doc Templating - #160
Conversation
…tainers instead of creating new images for each attack + program pair. This is experimental and current breaks multi-worker setups, but in single-worker scenarios results in significant speedups: - ~4x speed improvement when the base program image has already been built. - ~20-30% speed improvment when the base program image must be built. This also adds some timing code around container setup / creation.
…to grading containers. The workers now create a worker-specific temporary subdirectory in the worktmp volume to store their info inside of. This temp subdir is mounted into the grading container using docker subvolumes instead of directly mounting the worktmp volume. This requires some jank atm because the python docker sdk doesn't support docker volumes in its current release (`main` does have support, however). I ended up manually poking at the 'VolumeOptions' entry in the docker.types.Mount type to manually add the Subpath option. afaict this works fine and is basically what py docker sdk update for subvolumes does.
…hose in the current Piplock file (latest versions at time of writing).
…scripts for purging all images/containers.
… with cleaning up the work directory (need to play with this more, but I'm moving dev machines).
…esults table, this looks a bit weird now but before it was basically unreadable in dark mode(white text on white background).
… dulwich requires a hash function be provided.
…y other things) require access to the app context, but afaik there's no way to grab the context returned by create_app.
Previously, only some links used relationships while others used hand-written SQL statements to query linked objects. Trying to link these together with relationships results in fun circular dependency issues, which I'm guessing is why the raw sql statements were used. This partly gets around the circular dependency bs by combining the Attack and Result classes into a single file, I might undo this later if I can find another solution.
… program and args as the entrypoint instead of xargs with the target program. This allows preserving the return code from the target program.
…lating links to most pages. Previously docs had to hardcode links to different pages, which didn't work out well when the packs were used on different machines w/ different names.
… fixes the circular dep issue.
|
Thank you for this! I'll trust you to merge it, although @ChandlerSwift -- do you have any comments or would you like to review anything? |
…mplement them using sqlalchemy orm constructs)
|
Fixed an issue that stemmed from me not understanding how results were stored. Everything should be fine now I think. Also I'll note that there's a shell script injection vuln in the worker currently (https://github.com/UMDLARS/dtanm/pull/160/changes#diff-02bc36f026ba06349896b7dfd11078335296c083b61c7e427849b4fc2a2ca179R87) because it's effectively pasting the command args from the args file into a shell command. Anything attempted with that is contained inside the testing container where people can already use I initially tried xargs which doesn't have this problem, but it doesn't seem to preserve the return value from the executed command :/ |
Add current_results property.
I made an attempt at fixing this, injecting shell script doesn't appear to work anymore |
2 participants
This mostly update the dependencies to their ~latest versions and updates code as required or where nice.
This also restructures the way containers are created by the worker(s), instead of creating a new container for each program commit then for each attack, a container is created for each program commit then attack-specific information is sideloaded into that container using docker subvolumes. Test times are similar the first time a new program commit is tested (~6.5 sec -> ~4.5 sec), but subsequent runs against the same commit are significantly faster than before (~2.5 sec -> ~0.5 sec).
Lastly, this implements basic templating for the pack docs displayed on the Instructions (uses jinja). atm it only supports the following URL placeholders:
Previously, packs hardcoded URLs (e.g., calc has links to
hacks.lars.d.umn.edu), this allows replacing those with{{DTANM_LINK_xxx}}to get a link to page xxx for the current dtanm instance.