From f7ebe7e6ec6c629dc70836e0fa2a20f929519f38 Mon Sep 17 00:00:00 2001 From: Joshua S Brown Date: Wed, 24 Sep 2025 09:48:03 -0400 Subject: [PATCH 1/5] Update devel from staging (#1649) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Update issue templates * Update issue templates * Update pull_request_template.md (#1433) * [Release] v3.13.01 Resolve Master Branch Diff (#1485) * Address formatting of jquery file * add more descriptive posix path error message * Remove redundant password file creation command * Add eslint plugin for jsdoc to github action * Update esling.config.js to error if invalid jsdoc * Add missing eslint jsdoc plugin * Remove non existant rule * Fix jsdoc issues in data_router * Fix JSDOc issues in process.js fiel * Fix JSDOc issues in repo_router.js * Address JSDoc in tasks.js * Address JSDoc linting issues * Switch to ignores statement * Address more eslint jsdoc items * Be more selective about eslint rules * Remove requrie description because seems to be faulty * Address finicky jsdoc linter * Fix js formatting * Feature DLT 1109 foxx new token data (#1194) * Stub logic for updating related edge in router function for token/set * Include logic for conditionally updating token data and pushing data to edge. * Change token_key to remove illegal characters * Fix object attribute call * Formally add query params to function. Extract query params near start of action. Check both exist before taking action on edge. Update comments, notes, TODOs * Stub testing for user_router * Fix naming of user_router test so it will be picked up with sane specification. Add test to CMakeLists * Add some fixtures for creating related user to allow for integration testing * Fix manifest json. Hard code user collection check. Set time variable. * Use correct path for router to user endpoints. Extract some query params for validation. * Remove intentionally failing test. Break now working test into logical sections. Update comments. * Add test for new features * Add further validation of no unexpected side-effects * Remove extraneous user document checks and comments * More testing for branches in introduced logic. Additional check on newly inserted globus_coll * assert -> expect * Work around multiple access and ordering issue by utilizing more test user fixtures. * Rearrange key for token edge, add suggested fields to documents * Pull user_doc from existing logic. Use object destructuring. Add error condition. Introduce protobuf enum to backend. Add parser for other_token_data string. Update document creation functions * Extract fixture build call to separate script. Add script as fixture in cmake * Remove some extraneous logic, remove password file overwrite. * Remove unused zeromq flag * Adjust testing to include enum, expect error on invalid input combination, typo, naming * Adjust scope string in user_router test. Add validations to parseOtherTokenData. Add tests for positive and error cases of parseOtherTokenData. * Change chai expects in support.test.js to use callable. Update perms on fixture setup script. Use URL encoded pipe character in user_router.test.js * Group OR in user_router.js * Adjust user_router.test.js integration tests to access deconstructed other_token_data * Add baseline jsdoc for parse method. Remove some unwanted data from globus_token doc insert * Add GLOBUS_DEFAULT to AccessTokenType enum. Add default to user_router endpoint param. Update logic to switch based on enum. Test new change by specifying default type and other_token_data. Fix name of test. * Move query parameter validations earlier in function. Decrease nesting of switch logic. * Cleanup some whitespace changes, add desired comments on assumptions. * Formatting * Fix naming for support test so they will run. * Update CHANGELOG.md with changes in minor feature. Update comments on user_router.js for saving a new globus_coll * Add comment about more fields in support.js::parseOtherTokenData * Fix issue for formatter * Add expiration check * Commit for CI * Formatting --------- Co-authored-by: Anthony Ramirez * Add ci jobs to improve security posture * Change to unit stage for testing * Remove sast job * Attempt to fix semgrep sast job failure * Debug jobs * Add test stage for sast template job * Add missing test stage * Add dependency scanning * Split sast into separate test stage * Remove commented out line * Remove extra comment that is unused * Debugging log failure * Address jsdoc and typo in description * Remove outdated comments * Make comments JSDoc compliant * Address formatting inconsistency * Account for multiple images returned in log * Update .gitlab/common.yml Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Debugging log container matching * Fix typo in gitlab anchor * Debug escape anchor * Debug * Debug anchor 2 * Debug anchor 3 * Debug anchor 4 * Debug anchor 5 * Debug 6 * Debug 7 * Debug 8 * Fix names of containers from gcs-authz to gcs * Cleanup add comment back in * Update common.yml * Remove Exports * Update data_router.js Remove nested docs. * Update repo_router.js * Update task_router.js * Update eslint.config.js * Update eslint.config.js * fix jsdoc comment in posix path * Address JSDOc complaints * Address JSDoc errors * Address linter * Address linter errors * Fix order of parameters * Apply formatting * Add missing argument form strategy method call * Apply prettier * Update posix_path.js * Address ai suggestion about better error message * Debugging log failure * Account for multiple images returned in log * Update .gitlab/common.yml Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Debugging log container matching * Fix typo in gitlab anchor * Debug escape anchor * Debug * Debug anchor 2 * Debug anchor 3 * Debug anchor 4 * Debug anchor 5 * Debug 6 * Debug 7 * Debug 8 * Fix names of containers from gcs-authz to gcs * Cleanup add comment back in * Update common.yml * Remove Exports * Cleanup comments and apply formatting * Update nlhoman json version * Update protobuf versions * Add tests for authz router * Specify ubuntu 20.04 explicitly * Revert dependency versions * Fix problems with path consistent method * Cleanup javascript * Format javascript * Feature DLT 1127 relay new token data (#1217) * Add additional params to DatabaseAPI::userSetAccessToken to pass message with addtional info on to database API. * Add convenience method with old type signature of DatabaseAPI::userSetAccessToken for compatibility * Conditionally add other token options param when calling dbGetRaw. * Pass token type along to Database API. * Pass token type from message. * Extract logic for building URL from dbPost and dbGetRaw into separate function for easier testing and debugging * Extract url building logic from dbGet * Add some comments for moved code, address a TODO, add new TODO for AccessToken Message type field * Address default on token type, formatting * Add new AccessTokenType entries for a sentinel value and Globus default in SDMS.proto, use as desired in DatabaseAPI.cpp * Change query param to match backend changes * Restore formatting of SDMS.proto * Make built URL const where appropriate in DatabaseAPI.cpp and DatabaseAPI.hpp * Change default AccessTokenType for UserSetAccessTokenRequest in SDMS_Auth.proto. * Remove logging of potentially senitive data * Adjust AccessTokenType ACCESS_SENTINEL to have value of uint8 max. --------- Co-authored-by: Anthony Ramirez * [DLT-1110] Update files with prettier [DLT-1110] Add controller unit tests [DLT-1110] Remove debug [DLT-1110] Correct import statements, endpoint list [DLT-1110] Refactor out into MVC [DLT-1110] Add debug [DLT-1110] Functioning modal, needs refactoring [DLT-1110] Refactor dlg start transfer using OOO programming. Should be MVC cased on what I'm seeing but we'll see * [DLT-1110] Split branches * [DLT-1110] Update model, make logic private, remove controller form transfer request when update or get * [DLT-1110] Add commnets to logic * [DLT-1110] Correct HTML escape, francy tree init * [DLT-1110] Pull out template HTML * [DLT-1110] Fix update bug * [DLT-1110] Dependency injection * [DLT-1110] Update if statements * [DLT-1110] Update style * Revert "[DLT-1110] Mapped Collection Endpoint Browse (1/4)" * Address eslint * Address remaining eslint items * Fix bug on task_router.js abort function (#1234) Co-authored-by: Anthony Ramirez * Address Aaron feedback * applied potential fix to deprecation warning * Reorder createRecord * Apply formatting * Throw error if OpenStack error code returned from API * Add dependency between logs * Add dependencies on end to end jobs as well * Job rule is not supported * Remove always run log * add end to end signal job * Fix tag of signal * Feature DLT 1120 retrieve transfer token (#1228) * Add conditional switch and stub some logic for handling other resources in callbacks * Fix order of logging. Add redirect * Debug logging for web service. Attempt to set token. * Remove incorrect method call to data, use data attribute. * Add some TODOs and logging * Give setAccessToken ability to accept additional params to pass to protobuf * Add additional params to DatabaseAPI::userSetAccessToken to pass message with addtional info on to database API. * Add convenience method with old type signature of DatabaseAPI::userSetAccessToken for compatibility * Conditionally add other token options param when calling dbGetRaw. * Pass token type along to Database API. * Pass token type from message. * Stub logic for updating related edge in router function for token/set * Include logic for conditionally updating token data and pushing data to edge. * Change token_key to remove illegal characters * Fix object attribute call * Prettier formatting for datafed-ws.js * Roll back changes to DatabaseAPI.hpp and DatabaseAPI.cpp that are covered in #1127 * Refactor new inclusions in datafed-ws.js to use as much existing code as possible, only diverge where necessary. * datafed-ws.js change scopes to scope * datafed-ws.js log user ID array * datafed-ws.js clarify uid assignment at ui/authn endpoint, add note on error cases, remove incorrect comment about uid, add appropriate error if collection_id not present * datafed-ws.js Extract AccessTokenType enum, add comment about fetching from protobuf; remove unnecessary commented code * datafed-ws.js Extract transfer set logic when calling setAccessToken to improve readability of router. * datafed-ws.js Create function for resolving token type; create function to handle logic in building optional_data for setAccessToken; reduce unneccessary nesting in ui/authn endpoint * datafed-ws.js Fix spacing, fix typo in variable name * datafed-ws.js Fix bug referring to request session * datafed-ws.js Nesting was necessary. * datafed-ws.js Address some TODOs, set token type with more context * datafed-ws.js Remove some additional extraneous logging * datafed-ws.js Formatting * Add comment about token set * datafed-ws.js Update deprecated substring method, Fix order for error case when new user receives transfer token * TokenHandler.js Refactor token handling logic out of web server main. datafed-ws.js User new OAuthTokenHandler class to handle token logic. * TokenHandler.js Implement token validator; move OAuthTransferToken def to top. TokenHandler.test.js Minimal testing of token handler * TokenHandler.test.js Add tests for case when resource server is auth * TokenHandler.test.js Add testing for Globus transfer resource server * TokenHandler.js Implement validation for existence of required keys. TokenHandler.test.js Adjust other_tokens fixtures to be nested according to incoming data; formatting. * datafed-ws.js formatting * TokenHandler.js Update error messages, remove unnecessary code in getTokenType, returns in resolveTokenType * datafed-ws.js Add error handling around token_handler construction; add errors and handling for setAccessToken, JSDoc. * datafed-ws.js Remove hard coded collection_id. * datafed-ws.js Remove thrown errors in nested functions; single redirect. --------- Co-authored-by: Anthony Ramirez * Bug daps 1243 datafed web logging (#1251) * datafed-ws.js Reverse order of LogLevel object properties * datafed-ws.js Throw error so stack trace is adequately populated * Revert line number * TokenHandler.js Fix linting errors. --------- Co-authored-by: Anthony Ramirez * [DLT-1110] Mapped Collection Endpoint Browse (1/4) (#1240) * [DLT-1110] Update style [DLT-1110] Update if statements [DLT-1110] Dependency injection [DLT-1110] Fix update bug [DLT-1110] Pull out template HTML [DLT-1110] Correct HTML escape, francy tree init [DLT-1110] Add commnets to logic [DLT-1110] Update model, make logic private, remove controller form transfer request when update or get [DLT-1110] Split branches [DLT-1110] Update files with prettier [DLT-1110] Add controller unit tests [DLT-1110] Remove debug [DLT-1110] Correct import statements, endpoint list [DLT-1110] Refactor out into MVC [DLT-1110] Add debug [DLT-1110] Functioning modal, needs refactoring [DLT-1110] Refactor dlg start transfer using OOO programming. Should be MVC cased on what I'm seeing but we'll see * [DLT-1110] Mapped Collection Endpoint Browse Tests (2/4) (#1207) * [DLT-1110] Add tests * [DLT-1110] Update chai with mocha, attempt usage of mock-import * [DLT-1110] Add mocha config, correct prettier * [DLT-1110] Update tests, correct logic * [DLT-1110] Update tests, add fixtures, update setup, lock in packages * [DLT-1110] Update test * [DLT-1110] Config * [DLT-1110] Upgrade node v14 -> v23 * [DLT-1110] Upgrade node v23 -> v18, better jump * [DLT-1110] Correct ver .sh file * Revert "[DLT-1110] Correct ver .sh file" This reverts commit 5c78c59ea79c031452b4fca0179204e0e485b310. * Revert "[DLT-1110] Upgrade node v23 -> v18, better jump" This reverts commit 9d66847ca2ead4bca6f331c93ba206ff9c7695a6. * Revert "[DLT-1110] Upgrade node v14 -> v23" This reverts commit fb8b93d2137035500c138f0ac5f84f0faf35e958. * Revert "[DLT-1110] Config" This reverts commit 82715f07140b039eaf086c768b8ea281ade3da3d. * Revert "[DLT-1110] Mapped Collection Endpoint Browse Tests (2/4) (#1207)" This reverts commit 22545fa1927a0793d49e8172536a74ad0729e8d7. * added updated version t0 requirements.txt * Fix edge case in GridFTP Authz where '/' is used * cpp-py-formatter * Add changelog comment * Address prettier formatting * Add method for testing config, fix bug in global config, fix redundant log statements * Make error detection clearer in bool * Add mock core server * Add integration and, liveness test for mock * Address mock integration setup and consolidate * Apply formatting * Remove unused timeout variable from mock * Address JSDoc complaints * Apply prettier * Fix formatting and eslint prettier issues * Address codacy feedback * Add changelog comment * Clean up grammar in comment * Add note in Config.h * Apply clang formatting * Address codacy issues * Add env var * Update Arangodb packaged version to using 3.12.4 * Fix confusion in key uses in end to end tests * Add change log comment * Add changelog comment * corrected imported member * Add compose entries for foxx tests * Update the schema documentation to improve the usability. * Add example schema with a reference * Add notebook for linting jupyter * Using working directory option * Update .github/workflows/jupyternotebook-lint.yml Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Update python notebooks * Attempt to fix jupyternote book linting errors * Format with black * Specify line length with black to be consistent with flake8 * Add a Changelog comment * Change line length to 88 * Add black formatters * Make formatting consistent across project * Fix linter * Temporarily keep line length in black to 79 and address in separate issue * Change line length to 88 for now : * updated changelog * Fix bug in CI scripts associated with repo and gcs image build * Merge hotfixes into devel (#1326) * [DAPS-DEPS] Update cpp-py-formatter to 0.3.0 * [DAPS-DEPS] Update cpp-py-formatter to 0.3.1 * [DAPS-DEPS] Update ver, add push * updated versions for release * Update project.rst * Update format-check.yml * Pull in version numbers appropriately * Avoid build cache * Aggressive prune * Trigger rebuild * generate_repo_config.sh Accept env variable for datafed server from repo service instead of hard coding * Allow core config thread options to be read in from env * Add changelog comment * Adjust deployment version number * Fix Version month * Update CHANGELOG.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --------- Co-authored-by: Aaron Perez Co-authored-by: Blake Nedved Co-authored-by: Joshua S Brown Co-authored-by: JoshuaSBrown Co-authored-by: Anthony Ramirez Co-authored-by: nedvedba <145805866+nedvedba@users.noreply.github.com> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Feature DAPS 1215 foxx UserGetAccessTokenRequest mapped collection support (#1284) * Begin prototyping required code for mapped collection token retrieval. * SDMS_Auth.proto Add optional fields to UserGetAccessTokenRequest and UserAccessTokenReply to support finding correct tokens * user_router.js Add baseline query params and checks. DatabaseAPI.cpp & DatabaseAPI.hpp Use updated names and types. ClientWorker.cpp Update comments. * user_router.js Add logic for finding token based on collection; add token match and missing match cases * Return necessary data from Database to refresh tokens. Mock data on dir/list to test out changes. * user_router.js Add check for existence of Globus collection * user_router.js Remove existence check, move to filter method; Add notes on covered cases; add notes on code * datafed-ws.js Temporary handling of need consent response. * user_router.js Add queryParam validation for new params via joilib. ClientWorker.cpp Add refresh token try/catch to send through flow if refresh fails * datafed-ws.js Use correct converted casing for protobuf messages; redirect to error when needs_consent * datafed-ws.js No redirect, show error directly in list. * datafed-ws.js Early return to prevent further requests * support.js Update AccessTokenType enum. user_token.js New class introduced for handling logic for validating params when getting tokens and building GetAccessToken responses. user_router.js Replace logic with calls to new lib * user_token.js Add jsdoc definition for return type and format function. user_router.js Remove unnecessary comment. * user_token.js More jsdoc definition; Add validation for collection token document required fields. user_token.test.js Add preliminary unit tests for static methods. * CMakeLists.txt Add new unit tests to build. user_token.test.js Modify naming to match convention. * CMakeLists.txt Fix variable get. user_token.js Fix export. * user_fixture.js Add new user for get/token tests. user_router.test.js Add simple tests for get/token endpoint. user_token.js Fix naming for token document scopes field. user_token.test.js Changes for scope field bugfix. * user_router.js Formatting; Throw error if more than one token matches, add note about token type. * ClientWorker.cpp Address some TODOs and extraneous comments. DatabaseAPI.cpp Remove TODOs. SDMS_Auth.proto Remove unused field. * user_token.js Update formatUserToken to always return object values. DatabaseAPI.cpp Add note on possibly missing field. user_token.test.js Update tests to check for new expected values. * user_router.js Clean up some comments for clarity. datafed-ws.js Formatting. * ClientWorker.cpp Rewrite comment regarding mapped token refresh. * user_token.js Formatting. * CHANGELOG.md update for feature * user_router.js Re-introduce accidentally removed `globus_collection` object. * CMakeLists.txt Remove comment about necessity * Update token lookup logic to use `byExample`, add test case for missing Globus collection. Add globus_coll fixture. * SDMS_Auth.proto Change UserAccessTokenReply needs_consent to optional field. * user_router.js Update token/get to default to start consent flow when collection tokens are present. user_router.test.js Update test from error to needs_consent. * Formatting changes. --------- Co-authored-by: Anthony Ramirez * [DLT-1110] Implement Consent Required Action (3/4 & 4/4) (#1242) * [DLT-1110] Refactor browse into component and use CSS * [DAPS-1110] Update * [DAPS-1110] Update web server * [DAPS-1110] add refactors * [DAPS-1110] Prettier * [DAPS-1110] Refactor tests * [DAPS-1110] Remove .. in ep ls, remove logs * [DAPS-1110] Add sinon, remove dead code * [DAPS-1110] Address transfer start button bug * [DAPS-1110] eslint, update api.test.js to reflect usage * [DAPS-1110] format * [DAPS-1110] Correct bug * JSON parse error fix (#1328) * DatabaseAPI.cpp Pull in json dependency, use json object to serialize request payload * Replace json serialization in more locations where parseSearchRequest is being called. * Add comments around locations that will need json serialization * Convert missed body append to serialization format * typo * Refactoring DatabaseAPI.cpp to incorporate json serialization * More serialization work * Pull in changes from 1214 for dataPut * Bring in changes from 1214 for dataGet * DatabaseAPI.cpp Fix error with missing curly brace. Some formatting that is likely wrong. Serialize up to original point. Need to validate. * Add missing scope resolution * More typos * More typos. Scoping * Remove non-existant vars * declare body * Formatting * Formatting and verification of updates * Replace most places * Formatting * Remove redeclaration * DatabaseAPI.cpp Remove some comments. * DatabaseAPI.cpp Add missing push_back * Adding braces * Prevent Double Escaping string, by using nlohmann json parse to read in value. * remove escapeJSON calls, redundant with nlohmann json dump * Allow parsing for metric data to be run in parallel for now, output messages when old serialization is used. * Ensure ASCII setting. * Upgrade debug to warning. --------- Co-authored-by: Anthony Ramirez Co-authored-by: Austin Hampton Co-authored-by: Joshua S Brown * Feature - Mapped Collection Token Support (#1334) * Begin prototyping required code for mapped collection token retrieval. * SDMS_Auth.proto Add optional fields to UserGetAccessTokenRequest and UserAccessTokenReply to support finding correct tokens * user_router.js Add baseline query params and checks. DatabaseAPI.cpp & DatabaseAPI.hpp Use updated names and types. ClientWorker.cpp Update comments. * user_router.js Add logic for finding token based on collection; add token match and missing match cases * Return necessary data from Database to refresh tokens. Mock data on dir/list to test out changes. * user_router.js Add check for existence of Globus collection * user_router.js Remove existence check, move to filter method; Add notes on covered cases; add notes on code * datafed-ws.js Temporary handling of need consent response. * user_router.js Add queryParam validation for new params via joilib. ClientWorker.cpp Add refresh token try/catch to send through flow if refresh fails * datafed-ws.js Use correct converted casing for protobuf messages; redirect to error when needs_consent * datafed-ws.js No redirect, show error directly in list. * datafed-ws.js Early return to prevent further requests * support.js Update AccessTokenType enum. user_token.js New class introduced for handling logic for validating params when getting tokens and building GetAccessToken responses. user_router.js Replace logic with calls to new lib * user_token.js Add jsdoc definition for return type and format function. user_router.js Remove unnecessary comment. * user_token.js More jsdoc definition; Add validation for collection token document required fields. user_token.test.js Add preliminary unit tests for static methods. * CMakeLists.txt Add new unit tests to build. user_token.test.js Modify naming to match convention. * CMakeLists.txt Fix variable get. user_token.js Fix export. * user_fixture.js Add new user for get/token tests. user_router.test.js Add simple tests for get/token endpoint. user_token.js Fix naming for token document scopes field. user_token.test.js Changes for scope field bugfix. * user_router.js Formatting; Throw error if more than one token matches, add note about token type. * ClientWorker.cpp Address some TODOs and extraneous comments. DatabaseAPI.cpp Remove TODOs. SDMS_Auth.proto Remove unused field. * user_token.js Update formatUserToken to always return object values. DatabaseAPI.cpp Add note on possibly missing field. user_token.test.js Update tests to check for new expected values. * user_router.js Clean up some comments for clarity. datafed-ws.js Formatting. * ClientWorker.cpp Rewrite comment regarding mapped token refresh. * user_token.js Formatting. * CHANGELOG.md update for feature * user_router.js Re-introduce accidentally removed `globus_collection` object. * CMakeLists.txt Remove comment about necessity * Add collection_type and collection_id to dat/put task state * support.js Update getAccessToken method to work based on collection information, JSDoc. tasks.js Include context required in call to getAccessToken, add comment on refresh. TaskWorker.cpp Identify where changes will be reflected. * Update token lookup logic to use `byExample`, add test case for missing Globus collection. Add globus_coll fixture. * SDMS_Auth.proto Change UserAccessTokenReply needs_consent to optional field. * user_router.js Update token/get to default to start consent flow when collection tokens are present. user_router.test.js Update test from error to needs_consent. * Formatting changes. * SDMS_Auth.proto Add new fields to DataGetRequest * Begin refactor of token decision logic. * Address bugs in fetching and mapping data, make unit tests pass. * globus_token.js Match other models for get * tasks.js Pull in new logic for determining tokens, replace tokens; Needs to be verified. user_token.js Add new format function for compatibility with transfer doc replacement. * DatabaseAPI.cpp Sane serialization of json body for taskInitDataPut. TaskWorker.cpp Comments around work items. * DatabaseAPI.cpp Sane serialization of json body for taskInitDataGet. * ClientWorker.cpp Stub out required changes for consent flow when creating tasks * ClientWorker.cpp More thorough consent flow for procDataPutRequest, stub on needs_consent * tasks.js Throw error when initializing data put * tasks.js Format error throwing. ClientWorker.cpp comment out potentially unnecessary code. * user_token.js Add exists method to abstract check. tasks.js User exists method when checking token. * user_token.js JSDoc typing * tasks.js Add more required fields to taskRunDataPut for refresh in TaskWorker. * tasks.js Add collection_id to params in taskRunDataPut for refresh in TaskWorker. TaskWorker.cpp Refresh conditionals * DatabaseAPI.cpp Update merged devel changes to address some leftover code. * DatabaseAPI.cpp Add collection specification to get request. * DatabaseAPI.cpp whitespace. ClientWorker.cpp Clean up unused code. user_router.js Switch to checking token existence through standard API. * user_token.js Add comment about type return. data_router.js Do validations at router level for user collection token relationship. tasks.js Remove validation from logic level; make dataGet mirror dataPut; Updates to naming and params structure. * ClientWorker.cpp Revert to minimize diff. TaskWorker.cpp Remove unnecessary TODO; move set client for refresh in transfer tasks. * models/ JSDocs, formatting, make token fetches follow defined model. user_token.js JSDocs and typing. * user_token.js Cover case where collection does not exist. * Small changes for UI detection of collection type * fix epview call for UI changes, add list view change to react appropriately to needing consent according to current code * extract ep view logic to function for either path to utilize * small changes and notes for UI updates * UI search based on path, not ID. camelCasing * TaskWorker.cpp small fix to only enforce requirement of additional refresh fields when token is transfer token. * add params for dat/get on UI * task.js Fix bug where client is incorrectly referenced. * Add user model tests. Extract DataFedOAuthToken.js from user_token.js for reusability. * Add globus collection model tests. * Add globus token tests. Add fixtures to support globus token testing. Make all model properties static for iteration purposes. Utilize constructors for typing. * Add user token tests. Make non-existent user throw an error when building user tokens. * Change error to be not found * Formatting changes * Remove unused changes in support.js getAccessToken * Changelog update, update comments, remove log. --------- Co-authored-by: Anthony Ramirez * Add small fix for storeCollectionId (#1341) * Add small fix for storeCollectionId * move todo comment --------- Co-authored-by: Anthony Ramirez * Fix bug with playwright (#1343) * Fix bug with playwright install for end to end web test --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * [DAPS-1330] Address Download Extension Transfer (#1340) * [DAPS-1337] Refactor GET and Put mode * [DAPS-1337] Clear consent issue * [DAPS-1337] Correct bug * [DAPS-1337] Add Dir mode path logic update * Parse md json strings (#1347) * Parse md json strings * parse execQuery params * move to direct only * make params an object * remove console log --------- Co-authored-by: Anthony Ramirez * [DAPS-1349] Provenance Capture Update Bug (#1350) * [DAPS-1349] Return logic to original check * [DAPS-1349] version * [Release] February 2025 (#1352) * updated versions for release * Trigger rebuild * Adjust deployment version number * Fix Version month * Update Version.cmake * removed leading zero --------- Co-authored-by: JoshuaSBrown Co-authored-by: Joshua S Brown * Add .venv/ to gitignore (#1335) * [DAPS-1388] Add GCP Support (#1392) * [DAPS-1388] Add endpoint model, Add endpoint model, unify seperate collection logic, endpoint validation check, Add endpoint query validations, next is token * [DAPS-1388] Add tests for model * [DAPS-1388] PR comments * Update web/static/components/transfer/transfer-endpoint-manager.js Co-authored-by: Joshua S Brown --------- Co-authored-by: Joshua S Brown * Add missing python module (#1404) * Set reasonable default that will help with filling in repo form for core t… (#1400) * Set reasonable default that will help filling in repo form for core to repo communication in compose file * Update scripts/compose_generate_env.sh Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Set appropriate default when building compose image --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Add documentation describing how to test the foxx microservices unit … (#1393) * Add documentation describing how to test the foxx microservices unit tests * Update testing.rst * Note on database version, variable definition and usage when running arangodb version described, and steps to run unit test numbered. * Fix ci build (#1390) * updated requirements * updated requirements * updated protobuf submodule * update unit test ci version * updated protobuf version * updated setuptools version * fixed condition * changed where packages are installed * used python3.11 to create the virtual environment instead of latest * reverted protobuf version * testing specificying version * removed pyvenv in ci job for consistency * installed python3.11 in dependencies docker file * added missing -y flag * removed unification of install scripts * added missing python3.11 install * updated ci job to install python3.11 * changed installed python version to 3.9 since protobuf does not officially support 3.11 * changed installed python version to 3.9 since protobuf does not officially support 3.11 * missed version * updated includes for newer gcc versions * updated versions * updated nlohmann json * updated json schema version * changed library search path * hopefully fixed json dep issues * updating function signature to match newer version of base class * updated protobuf * updated protobuf and cmake * downgraded cmake * fixed protobuf version * fixed protobuf version * 1398 pin python version (#1405) * Pin python version * fixed incorrect path * Switch to using .tar file install of libsodium (#1414) * 1413 libsodium build refactor (#1415) * Switch to using .tar file install of libsodium * Switch bad option in wget command from -C to -P * Update scripts/dependency_install_functions.sh * Update scripts/dependency_install_functions.sh * Update dependency_install_functions.sh Libsodium folder version number. * Update dependency_install_functions.sh Make paths explicit. * cleaned up comments * Update scripts/dependency_install_functions.sh Co-authored-by: Joshua S Brown * Update cmake/JSONSchema.cmake Co-authored-by: Joshua S Brown * Update scripts/dependency_install_functions.sh Co-authored-by: Joshua S Brown * Update cmake/JSONSchema.cmake Co-authored-by: Joshua S Brown * add changes from review * Changed install scripts to be consistent with new python installation method (#1417) * changed install scripts to be consistent * added sudo check * add apt sources check --------- Co-authored-by: Blake Nedved Co-authored-by: nedvedba <145805866+nedvedba@users.noreply.github.com> * Fix regression (#1428) * Fix docker logs output in end-to-end tests (#1426) * Improve serialization randomization (#1427) * Increase serialization randomization pause heuristic - 6 seconds is not long enough to register running pipelines. * improve serialization * Upgrade formatter (#1440) * Update action for CPP and Python formatter * 1437 fix zmq inproc bind order (#1438) * Reorder zmq bind and connect calls to prevent undefined behavior when using INPROC * Make sure zmq messages are closed when exception is encountered * Make correction to socket name * Initialize cmd call * Add missing include * Allow looping over containers if there is more than one. (#1446) * 1445 docker logs multiple containers (#1447) * Allow looping over containers if there is more than one. * Fix multipline comment * Fix conflicting dependency install flags (#1444) * [DAPS-1408] Provenance Visual Management Lint (1/2) (#1431) * [DAPS-14xx] Move files over and move out svg * [DAPS-1408] Provenance Visual Management (2/2) (#1419) * [DAPS-14xx] Base * [DAPS-14xx] Add styling and tooltips * [DAPS-14xx] Add node and label customization * [DAPS-14xx] Add editor modal on right-click. Draggable model * [DAPS-14xx] Revert * [DAPS-14xx] Seperate files, update styles * [DAPS-14xx] simplify feat * [DAPS-14xx] Expand mocha tests, prettier * [DAPS-14xx] Update checkout version * [DAPS-14xx] Add todo * [DAPS-14xx] Update tests, consts, fix tooltip * [DAPS-14xx] Base: [DAPS-14xx] Address collapse, expand, hide * [DAPS-14xx] THEME, address comments * [DAPS-14xx] Zoom * [DAPS-14xx] JSDOC * Update web/static/jquery-ui-dark/datafed.css Co-authored-by: Joshua S Brown * Update web/static/jquery-ui-dark/datafed.css Co-authored-by: Joshua S Brown * Update web/static/jquery-ui-dark/datafed.css Co-authored-by: Joshua S Brown --------- Co-authored-by: Joshua S Brown * [DAPS-1395] Address Memory Leaks in Core Server (#1424) * Add missing virtual destructors * Fix ZeroMQ INPROC and TCP context * Add memory tests to CMakeLists.txt as an option * Add protobuf library shutdown to unit tests for memory cleanup * [TASK] Update tests and fixture (#1454) * [DAPS-1331] Collection ID Updated on restart (#1468) * [DAPS-1331] Modified the script to only delete and recreate guest collections when the base path actually changes * [DAPS-1331] Add base path logic * [DAPS-1331] Update guest collection file * [DAPS-1331] Fix base path caching logic in should_recreate_guest_collection function to recreate by default when no cached path is available * [DAPS-1331] Add tests * [DAPS-1331] Revert Gitlab CI * [DAPS-RELEASE] * [TASK] Remove icons from labels (#1489) --------- Co-authored-by: Anthony <64566330+t-ramz@users.noreply.github.com> Co-authored-by: Anthony Ramirez Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: Aaron Perez Co-authored-by: AronPerez Co-authored-by: Blake Nedved Co-authored-by: par-hermes Co-authored-by: nedvedba <145805866+nedvedba@users.noreply.github.com> Co-authored-by: Austin Hampton Co-authored-by: Theo Beers <32523293+theodore-s-beers@users.noreply.github.com> * [DAPS-1625] - user id map for session key josh (#1644) * [DAPS-1625] fix bug in session and transient keys. --------- Co-authored-by: AronPerez * [TASK] Correct 2 spaces to 4 for YML (#1640) * fix: merge conflict changes that were missed. --------- Co-authored-by: Aaron Perez Co-authored-by: Anthony <64566330+t-ramz@users.noreply.github.com> Co-authored-by: Anthony Ramirez Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: Aaron Perez Co-authored-by: Blake Nedved Co-authored-by: par-hermes Co-authored-by: nedvedba <145805866+nedvedba@users.noreply.github.com> Co-authored-by: Austin Hampton Co-authored-by: Theo Beers <32523293+theodore-s-beers@users.noreply.github.com> --- core/database/tests/test_fixture_setup.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/core/database/tests/test_fixture_setup.sh b/core/database/tests/test_fixture_setup.sh index 4e299b7ea..b88b2c521 100755 --- a/core/database/tests/test_fixture_setup.sh +++ b/core/database/tests/test_fixture_setup.sh @@ -82,6 +82,7 @@ while [ : ]; do echo "Error: Invalid option" exit ;; + esac done From 813c06b3b3708759a7afad3063f7f3746eb9def6 Mon Sep 17 00:00:00 2001 From: Joshua S Brown Date: Wed, 24 Sep 2025 10:03:56 -0400 Subject: [PATCH 2/5] [DAPS-1585] [Dependencies] update: upgrade ssl dependency. 3.2.5 (#1646) --- external/DataFedDependencies | 2 +- repository/docker/Dockerfile.gcs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/external/DataFedDependencies b/external/DataFedDependencies index 77e13c73d..eee5cfb8d 160000 --- a/external/DataFedDependencies +++ b/external/DataFedDependencies @@ -1 +1 @@ -Subproject commit 77e13c73d55e8f7d1ed2c4b2b72f8737762a5314 +Subproject commit eee5cfb8dac44b806f2a78526603cb363bc2a002 diff --git a/repository/docker/Dockerfile.gcs b/repository/docker/Dockerfile.gcs index b84e035e9..06ba36cb4 100644 --- a/repository/docker/Dockerfile.gcs +++ b/repository/docker/Dockerfile.gcs @@ -169,7 +169,7 @@ COPY --chown=datafed:root ./repository/docker/entrypoint_authz.sh ${BUILD_DIR}/r USER root -ENV LD_LIBRARY_PATH=/opt/datafed/dependencies/lib/ +ENV LD_LIBRARY_PATH=/opt/datafed/dependencies/lib/:/opt/datafed/dependencies/lib64/ # Remove --client-id from command because it was deprecated RUN sed -i '/--client-id/d' /entrypoint.sh From ef8db2fef6df68875ec22e65543949a437203d87 Mon Sep 17 00:00:00 2001 From: Joshua S Brown Date: Wed, 24 Sep 2025 16:19:05 -0400 Subject: [PATCH 3/5] Staging to devel update (#1657) * Update devel from staging From 94a207ad32c46b8b3f2c6eda038f495f528302f8 Mon Sep 17 00:00:00 2001 From: Joshua S Brown Date: Wed, 24 Sep 2025 16:40:25 -0400 Subject: [PATCH 4/5] [DAPS-1605] fix install_foxx.sh - split ssl_args (#1623) * fix(install_foxx): remove ssl_args argument it is not a valid flag. --- scripts/install_foxx.sh | 4 ---- 1 file changed, 4 deletions(-) diff --git a/scripts/install_foxx.sh b/scripts/install_foxx.sh index 5d91e1ec2..edc98e6fb 100755 --- a/scripts/install_foxx.sh +++ b/scripts/install_foxx.sh @@ -55,7 +55,6 @@ local_DATABASE_USER="root" local_DATABASE_PORT="8529" if [ -f "${local_SSL_CERT_FILE}" ]; then - ssl_args="--ssl.cafile ${local_SSL_CERT_FILE}" export NODE_EXTRA_CA_CERTS="${local_SSL_CERT_FILE}" fi @@ -178,19 +177,16 @@ else "${local_ARANGOSH_SERVER_ENDPOINT_SCHEME}://${local_DATAFED_DATABASE_HOST}:${local_DATABASE_PORT}" \ --server.password "${local_DATAFED_DATABASE_PASSWORD}" \ --server.username "${local_DATABASE_USER}" \ - "${ssl_args}" \ --javascript.execute "${PROJECT_ROOT}/core/database/foxx/db_create.js" # Give time for the database to be created sleep 2 arangosh --server.endpoint "${local_ARANGOSH_SERVER_ENDPOINT_SCHEME}://${local_DATAFED_DATABASE_HOST}:${local_DATABASE_PORT}" \ --server.password "${local_DATAFED_DATABASE_PASSWORD}" \ --server.username "${local_DATABASE_USER}" \ - "${ssl_args}" \ --javascript.execute-string 'db._useDatabase("sdms"); db.config.insert({"_key": "msg_daily", "msg" : "DataFed servers will be off-line for regular maintenance every Sunday night from 11:45 pm until 12:15 am EST Monday morning."}, {overwrite: true});' arangosh --server.endpoint "${local_ARANGOSH_SERVER_ENDPOINT_SCHEME}://${local_DATAFED_DATABASE_HOST}:${local_DATABASE_PORT}" \ --server.password "${local_DATAFED_DATABASE_PASSWORD}" \ --server.username "${local_DATABASE_USER}" \ - "${ssl_args}" \ --javascript.execute-string "db._useDatabase(\"sdms\"); db.config.insert({ \"_key\": \"system\", \"_id\": \"config/system\"}, {overwrite: true } );" fi From d753773043c9c98ae33d5bad3c025128245acda0 Mon Sep 17 00:00:00 2001 From: JoshuaSBrown Date: Thu, 2 Oct 2025 13:39:38 -0400 Subject: [PATCH 5/5] update: DataFedDependencies to 57483e1cd4eac9d84162dd7202e72fe353728361 --- external/DataFedDependencies | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/external/DataFedDependencies b/external/DataFedDependencies index b3528638d..57483e1cd 160000 --- a/external/DataFedDependencies +++ b/external/DataFedDependencies @@ -1 +1 @@ -Subproject commit b3528638dc7e78633b60b607fd0227aa5862bf53 +Subproject commit 57483e1cd4eac9d84162dd7202e72fe353728361