diff --git a/.gitignore b/.gitignore index 24dac96..589a302 100644 --- a/.gitignore +++ b/.gitignore @@ -191,3 +191,5 @@ nbdist/ .github/** .gitignore /src/main/resources/webui/** +data/** +start.sh \ No newline at end of file diff --git a/LICENSE b/LICENSE index b8ec035..12d3f32 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2021-2024 Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. +Copyright (c) 2021-2026 Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.adoc b/README.adoc index 42d2265..effa19f 100644 --- a/README.adoc +++ b/README.adoc @@ -4,69 +4,326 @@ image:https://github.com/AxisAlexNT/HiCT_JVM/actions/workflows/autobuild-release == Launching pre-built version -**NOTE: currently only Windows (tested on 10 and 11) and Linux (with `glibc`, common Debain/Ubuntu are OK, Alpine users are out of luck) are supported, native libraries for MacOS are not bundled in these builds. Only AMD64 platform is supported. On Windows you might need to install https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170[additional libraries]**. -1. Install Java 19 or newer (older versions won't be able to launch this code); -1. Make sure that `JAVA_HOME` variable points to the correct installation path (if you have multiple JREs or JDKs); -1. Download latest "fat" JAR from the https://github.com/ctlab/HiCT_JVM/releases[*Releases* page] in *Assets* section. Latest build will usually be on top, however the most stable implementation is in the build from `master` branch (called "Latest autogenerated build (branch master)"). You can rename it to `hict.jar` for convenience; -1. Open a terminal and change directory to where the downloaded `hict.jar` is located; -1. Issue `java -jar hict.jar` command and wait until message `Starting WebUI server on port 8080 ... WebUI Server started` appears; -1. Open your browser and navigate to the `http://localhost:8080` where HiCT WebUI should now be available. +== For users of `.jar` distribution -=== Startup options +This section is intended for bioinformatics users who download a ready-to-run fat JAR from GitHub Releases. +You need to install Java 21+ (this project is built for Java 21 bytecode). +Download the latest fat JAR from the https://github.com/ctlab/HiCT_JVM/releases[Releases page] (Assets section). +**NOTE:** prebuilt native bundles are currently provided for *Windows* (tested on 10/11) and *Linux with glibc* (common Debian/Ubuntu-like distributions). Alpine/musl is not supported by these bundled binaries. Current prebuilt artifacts are AMD64-only. On Windows you might need to install https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170[Microsoft Visual C++ Redistributable]. -Currently, there are multiple environment variables that could be set prior to launching HiCT. +=== Quick start -* `DATA_DIR` -- should be a path to the directory containing `.hict.hdf5`, `.agp` and `fasta` files. These files could be anywhere in subtree of this directory, it is scanned recursively. -* `VXPORT` -- should be an integer between `1` and `65535` denoting port number which will be served by HiCT API. Note that listening on ports below `4096` usually requires some kind of administrative privileges. If not provided, the default value is `5000`. Startup might fail if the port is already occupied by another service. Be sure to set correct port in Connection -> API Gateway field in HiCT WebUI if changed. -* `WEBUI_PORT` -- should be an integer between `1` and `65535` denoting port number which will be served by HiCT WebUI. Note that listening on ports below `4096` usually requires some kind of administrative privileges. If not provided, the default value is `8080`. Startup might fail if the port is already occupied by another service. -* `SERVE_WEBUI` -- should either be `true` or `false` telling whether to start serving HiCT WebUI on the desired port or not. Might be useful during debugging or when WebUI is served by another process. Default is `true`. This option does not have any effect in case WebUI is not packed into the jar file. -* `TILE_SIZE` -- should be an integer greater than one. Defines the default tile size for visualization. Experimental setting, currently might break WebUI renderer. Default is `256`. The greater the tile size is, the less tiles are shown on screen and therefore less requests are sent to the server, but each request could potentially take longer to process. +1. Download the latest `-fat.jar` from the Releases page (Assets) and rename it to `hict-fat.jar`. +2. Place your `.hict.hdf5`, `.mcool`, `.cool`, `.agp`, and `.fasta` files under a single directory. +3. Run: ++ +```bash +java -jar hict-fat.jar start-server +``` ++ +Directory with files is set using `DATA_DIR` environment variable, by default it scans subtree of the directory in which `hict-fat.jar` is launched from. +In Linux you may set it as follows: ++ +```bash +DATA_DIR=/path/to/data/ java -jar hict-fat.jar start-server +``` ++ +4. Open WebUI at `http://localhost:8080`. + +=== CLI commands (summary) + +```bash +# API + WebUI (default mode, includes converters in WebUI as descibed below) +java -jar hict-fat.jar start-server + +# API only (no WebUI) +java -jar hict-fat.jar start-api-server + +# Convert .mcool -> .hict.hdf5 (CLI mode) +java -jar hict-fat.jar convert mcool-to-hict \ + --input /data/sample.mcool \ + --output /data/sample.hict.hdf5 + +# Convert .hict.hdf5 -> .mcool (CLI mode) +java -jar hict-fat.jar convert hict-to-mcool \ + --input /data/sample.hict.hdf5 \ + --output /data/sample.mcool +``` + +Get full CLI help: + +```bash +java -jar hict-fat.jar --help +java -jar hict-fat.jar start-server --help +java -jar hict-fat.jar start-api-server --help +java -jar hict-fat.jar convert --help +java -jar hict-fat.jar convert mcool-to-hict --help +java -jar hict-fat.jar convert hict-to-mcool --help +``` + +=== WebUI conversion (Experimental / W.I.P.) + +WARNING: WebUI conversion is experimental and may be slower or less stable than the CLI. + +1. Open the WebUI. +2. Use *File → Convert Coolers*. +3. Track progress in the conversion window. + +=== API access (Experimental / W.I.P.) + +WARNING: The API is still evolving. Endpoints, parameters, and response formats may change. + +Example (Python) for fetching a submatrix tile as an image: + +```python +import requests + +host = "http://localhost:5000" +params = { + "version": 0, + "bpResolution": 10000, + "format": "PNG_BY_PIXELS", + "row": 0, + "col": 0, + "rows": 512, + "cols": 512, +} -An example of launching HiCT with parameters: +r = requests.get(f"{host}/get_tile", params=params) +r.raise_for_status() +data = r.json() +png_data_url = data["image"] +print(png_data_url[:64]) +``` + +To apply visualization/normalization settings before fetching tiles: + +* POST `/set_visualization_options` with visualization parameters. +* POST `/set_normalization` with normalization settings. +* Then call `/get_tile` as shown above. + +=== Supported platforms / JDK details + +* *OS/CPU (prebuilt libs):* Linux (glibc) and Windows, AMD64. +* *Not bundled by default:* macOS variants and Linux ARM variants. +* *JDK:* Java 19 or newer is required for running/building this repository. + +== Startup options and CLI + +The fat JAR is runnable and exposes a CLI with subcommands: + +* `start-server` -- API + WebUI (default when no args are given) +* `start-api-server` -- API only (no WebUI) +* `convert` -- conversion tools +** `convert mcool-to-hict` +** `convert hict-to-mcool` + +Help: -==== *Linux, bash:* ```bash -DATA_DIR=/home/${USER}/hict/data SERVE_WEBUI=false java -jar hict.jar +java -jar hict.jar --help +java -jar hict.jar convert --help +java -jar hict.jar convert mcool-to-hict --help ``` -==== *Windows, cmd:* +Environment variables supported by the server startup: + +* `DATA_DIR` -- directory that is scanned recursively for `.hict.hdf5`, `.agp`, `fasta`, `.cool`, and `.mcool` files. +* `VXPORT` -- API gateway port, default `5000`. +* `WEBUI_PORT` -- WebUI port, default `8080`. +* `SERVE_WEBUI` -- `true`/`false`, default `true`. +* `TILE_SIZE` -- default visualization tile size, default `256`. +* `MIN_DS_POOL` / `MAX_DS_POOL` -- min/max pool sizes used when opening chunked datasets. + +=== Launch examples (fat JAR) + +==== Linux (bash) + +```bash +DATA_DIR=/home/${USER}/hict/data java -jar hict.jar + +# API only +DATA_DIR=/home/${USER}/hict/data java -jar hict.jar start-api-server + +# Explicit server (API + WebUI) +DATA_DIR=/home/${USER}/hict/data java -jar hict.jar start-server +``` + +==== Windows (cmd) + ```cmd set DATA_DIR="D:\hict\data" set WEBUI_PORT="8888" -java -jar hict.jar +java -jar hict.jar start-server ``` -==== *Windows, PowerShell:* +==== Windows (PowerShell) + ```powershell $env:DATA_DIR = "D:\hict\data" $env:WEBUI_PORT = "8888" -java -jar hict.jar +java -jar hict.jar start-server +``` + +==== Custom JVM options + +```bash +DATA_DIR=/home/${USER}/hict/data java -ea -Xms512M -Xmx16G -jar hict.jar start-api-server +``` + +=== Launch examples (Gradle, from source) + +```bash +# Default: runs HiCT CLI (equivalent to `java -jar ...`) +./gradlew clean run + +# Explicit modes +./gradlew run --args="start-server" +./gradlew run --args="start-api-server" ``` -==== Custom JVM Options +== Converter workflows (`.mcool` ↔ `.hict.hdf5`) -Of course, you can also pass JVM parameters like this: +=== CLI commands + +Use the JVM CLI for both directions: ```bash -DATA_DIR=/home/${USER}/hict/data SERVE_WEBUI=false java -ea -Xms512M -Xmx16G -jar hict.jar +# mcool -> hict +java -jar hict.jar convert mcool-to-hict \ + --input /data/sample.mcool \ + --output /data/sample.hict.hdf5 + +# hict -> mcool +java -jar hict.jar convert hict-to-mcool \ + --input /data/sample.hict.hdf5 \ + --output /data/sample.roundtrip.mcool ``` -=== Startup errors +=== Web conversion API flow -Since library naming conventions are different for different platform and libraries, there is currently a mechanism to try and load each library under a different name. This CAN produce errors on server startup, you can ignore them if `Starting WebUI server on port 8080 ... WebUI Server started` message appeared in console. +Typical asynchronous conversion sequence used by WebUI/integrations: -If, however, server works but maps are not displayed in WebUI and an error sign displays at the bottom right corner of WebUI, you should check console for error output. +1. *Upload*: `POST /api/convert/upload` + * Upload source file and target format metadata. + * Response returns a `jobId`. +2. *Status polling*: `GET /api/convert/status/{jobId}` + * Poll until state becomes `DONE` or `FAILED`. +3. *Download*: `GET /api/convert/download/{jobId}` + * Download converted artifact when status is `DONE`. -== Obtaining `.hict.hdf5` files +Recommended size limits: -Currently, it's necessary to use https://github.com/ctlab/HiCT_Utils[`HiCT_Utils` package] for the file format conversion, there are plans to simplify this process. +* Keep upload limits explicit at ingress/proxy and app gateway. +* For JVM safety, avoid unbounded request bodies in production; set max request size and timeouts. +* For very large matrices, prefer direct local file conversion (CLI) and then load resulting artifacts through `DATA_DIR`. -== Building `HiCT_JVM` from source +== Scaffolding API behavior notes + +Scaffolding operations are served as POST endpoints and return updated assembly information: + +* `/reverse_selection_range` +* `/move_selection_range` +* `/split_contig_at_bin` +* `/group_contigs_into_scaffold` +* `/ungroup_contigs_from_scaffold` +* `/move_selection_to_debris` + +Important tile-version expectation: -To start building from source, you can run: +* Tile requests use `GET /get_tile?...&version=`. +* If the requested version is *older* than server-side tile version, server returns HTTP `204` (no tile body) to force client invalidation. +* If the requested version is newer, server advances the internal version counter. +* Practical client rule: after each scaffolding mutation, increment your tile version and refresh visible tile requests. + +== Startup errors and JHDF5 native library troubleshooting + +During startup, you may see several native-library load attempts with warnings/errors. This can be expected because different platform-specific library names are tried. + +If startup completes and API/WebUI are healthy, these warnings can be non-fatal. + +When native loading actually fails: + +1. Confirm architecture match (AMD64 JVM + AMD64 native bundle). +2. Confirm OS compatibility (Linux glibc; not Alpine/musl). +3. On Linux, ensure native/plugin paths are discoverable, for example: ++ +```bash +export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/to/hdf5/lib:/path/to/hdf5/lib/plugin" +export HDF5_PLUGIN_PATH="/path/to/hdf5/lib/plugin" ``` +4. On Windows, install/update Visual C++ runtime redistributables. +5. Verify Java version (`java -version`) is 19+. +6. If tiles fail to render but server starts, inspect logs for `UnsatisfiedLinkError` and HDF5 plugin load failures. + +== Production checklist (short) + +Before deploying to production, verify: + +* Logging: structured logs, retention, and centralized collection. +* Metrics/health: request latency/error metrics and liveness/readiness checks. +* Limits: request body size, timeouts, and JVM heap sizing are set explicitly. +* Graceful shutdown: stop accepting traffic, finish in-flight requests, then terminate. +* Backup/cleanup: regular backup strategy for source/converted files and periodic cleanup of temporary/intermediate artifacts. + +== Building `HiCT_JVM` from source + +To build from source: + +```bash ./gradlew clean build ``` +=== Dependency management workflow + +This project uses Gradle dependency locking (`gradle.lockfile`) to keep transitive dependency resolution reproducible. + +* Refresh lock state after dependency changes: ++ +```bash +./gradlew dependencies --write-locks +``` +* Inspect the resolved version for a specific dependency before/after updates: ++ +```bash +./gradlew dependencyInsight --dependency org.slf4j:slf4j-api --configuration runtimeClasspath +./gradlew dependencyInsight --dependency ch.qos.logback:logback-classic --configuration runtimeClasspath +./gradlew dependencyInsight --dependency org.jetbrains:annotations --configuration compileClasspath +``` + +Commit both `build.gradle.kts` and `gradle.lockfile` together whenever lock state changes. + Current progress on modifying HDF5 and JHDF5 configuration resides in https://github.com/AxisAlexNT/jhdf5-with-plugins-configuration-snapshot[my personal repository]. Modified configuration is necessary to rebuild native libraries (HDF5, HDF5 plugins and JHDF5 should all be build as dynamic libraries). However, prebuilt native libraries for AMD64 Windows and Linux platforms are already present in `HiCT_JVM` repository. Missing platforms are Linux on `armv7` and `aarch64` and MacOS (both `amd64` and `aarch64` variants). + +== Conversion tools (CLI + API) + +A native converter module is now available in JVM codebase with two services: + +* `McoolToHictConverter` (`mcool-to-hict`) +* `HictToMcoolConverter` (`hict-to-mcool`) + +CLI launcher: + +```bash +./gradlew runConversionCli --args="convert hict-to-mcool --input=/data/sample.hict.hdf5 --output=/data/sample.mcool --resolutions=10000,50000 --compression=4 --chunk-size=8192" +./gradlew runConversionCli --args="convert mcool-to-hict --input=/data/sample.mcool --output=/data/sample.hict.hdf5 --resolutions=10000,50000 --parallelism=16" +``` + +Arguments: + +* `--input=` source file path +* `--output=` destination file path +* `--resolutions=` optional resolution filter +* `--compression=<0..9>` deflate level (`0` means chunked/no deflate) +* `--chunk-size=` chunk size for streaming traversal +* `--agp= --apply-agp` apply AGP before `hict-to-mcool` export +* `--parallelism=` max worker threads (default: available CPU cores) + +Web API endpoints: + +* `POST /convert/upload` (multipart + query params: `direction`, `resolutions`, `compression`, `chunkSize`, `applyAgp`, `agpPath`, `parallelism`) +* `GET /convert/jobs/:jobId` +* `GET /convert/download/:jobId` + +Conversion jobs are asynchronous, include streaming logs/error details, enforce upload size limit and have temporary file cleanup TTL. diff --git a/build.gradle.kts b/build.gradle.kts index 21c467e..cec2196 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -25,11 +25,13 @@ import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar import org.gradle.api.tasks.testing.logging.TestLogEvent.* import java.io.ByteArrayOutputStream import java.nio.file.Files +import java.nio.file.StandardCopyOption +import java.nio.file.StandardCopyOption.REPLACE_EXISTING plugins { java application - id("com.github.johnrengelman.shadow") version "7.1.2" + id("com.gradleup.shadow") version "8.3.9" } group = "ru.itmo.ctlab.hict" @@ -43,6 +45,12 @@ repositories { val vertxVersion = "4.4.1" val junitJupiterVersion = "5.9.1" +val slf4jVersion = "1.7.36" +val logbackVersion = "1.2.13" + +dependencyLocking { + lockAllConfigurations() +} val mainVerticleName = "ru.itmo.ctlab.hict.hict_server.MainVerticle" val launcherClassName = "io.vertx.core.Launcher" @@ -51,9 +59,13 @@ val watchForChange = "src/**/*" val doOnChange = "${projectDir}/gradlew classes" val versionFile = file("${project.projectDir}/version.txt") +val webUiPackageJson = file("${project.projectDir}/../HiCT_WebUI/package.json") val webUICloneDirectory = layout.buildDirectory.dir("webui").get() -val webUIRepositoryDirectory = webUICloneDirectory.dir("HiCT_WebUI") +val localWebUIRepositoryDirectory = layout.projectDirectory.dir("../HiCT_WebUI") +val remoteWebUIRepositoryDirectory = webUICloneDirectory.dir("HiCT_WebUI") +val webUIRepositoryDirectory = + if (localWebUIRepositoryDirectory.asFile.exists()) localWebUIRepositoryDirectory else remoteWebUIRepositoryDirectory val webUIRepositoryAddress = "https://github.com/ctlab/HiCT_WebUI.git" val webUITargetDirectory = layout.projectDirectory.dir("src/main/resources/webui") val webUIBranch = "dev-0.1.5" @@ -61,9 +73,11 @@ val webUIBranch = "dev-0.1.5" version = readVersion() application { - mainClass.set(launcherClassName) + mainClass.set("ru.itmo.ctlab.hict.hict_server.tools.HictCli") } +val lombokVersion = "1.18.42" + dependencies { // implementation(fileTree("src/main/resources/libs")) // runtimeOnly(fileTree("src/main/resources/libs/natives")) @@ -74,23 +88,19 @@ dependencies { // https://mvnrepository.com/artifact/cisd/base implementation("cisd:base:18.09.0") implementation("org.jetbrains:annotations:24.0.0") - implementation("org.jetbrains:annotations:24.0.0") // https://mvnrepository.com/artifact/org.apache.bcel/bcel implementation("org.apache.bcel:bcel:6.7.0") + compileOnly("org.projectlombok:lombok:$lombokVersion") + annotationProcessor("org.projectlombok:lombok:$lombokVersion") + testCompileOnly("org.projectlombok:lombok:$lombokVersion") + testAnnotationProcessor("org.projectlombok:lombok:$lombokVersion") - - compileOnly("org.projectlombok:lombok:1.18.22") - annotationProcessor("org.projectlombok:lombok:1.18.22") - testCompileOnly("org.projectlombok:lombok:1.18.22") - testAnnotationProcessor("org.projectlombok:lombok:1.18.22") - - - implementation("org.slf4j:slf4j-api:1.7.+") + implementation("org.slf4j:slf4j-api:$slf4jVersion") // implementation("org.slf4j:slf4j-nop:1.7.+") - implementation("ch.qos.logback:logback-classic:1.2.+") + implementation("ch.qos.logback:logback-classic:$logbackVersion") implementation(platform("io.vertx:vertx-stack-depchain:$vertxVersion")) @@ -121,6 +131,7 @@ dependencies { // https://mvnrepository.com/artifact/org.scijava/native-lib-loader implementation("org.scijava:native-lib-loader:2.4.0") + implementation("info.picocli:picocli:4.7.6") } @@ -133,7 +144,12 @@ java { tasks.withType { archiveClassifier.set("fat") manifest { - attributes(mapOf("Main-Verticle" to mainVerticleName)) + attributes( + mapOf( + "Main-Verticle" to mainVerticleName, + "Main-Class" to "ru.itmo.ctlab.hict.hict_server.tools.HictCli" + ) + ) } mergeServiceFiles() } @@ -145,7 +161,17 @@ tasks.withType { } } -tasks.withType { + + +tasks.register("runConversionCli") { + group = "application" + description = "Run conversion CLI (hict-to-mcool / mcool-to-hict subcommands)" + classpath = sourceSets["main"].runtimeClasspath + mainClass.set("ru.itmo.ctlab.hict.hict_server.tools.HictCli") +} + + +tasks.withType().configureEach { doFirst { environment( "LD_LIBRARY_PATH", @@ -161,13 +187,6 @@ tasks.withType { ) environment("VERTXWEB_ENVIRONMENT", "dev") } - args = listOf( - "run", - mainVerticleName, - "--redeploy=$watchForChange", - "--launcher-class=$launcherClassName", - "--on-redeploy=$doOnChange" - ) } @@ -232,6 +251,27 @@ tasks.register("buildWebUI") { dependsOn("cleanWebUI") doLast { try { + if (localWebUIRepositoryDirectory.asFile.exists()) { + println("Using local HiCT_WebUI checkout at ${localWebUIRepositoryDirectory.asFile.absolutePath}") + project.exec { + commandLine("git", "checkout", webUIBranch) + workingDir = localWebUIRepositoryDirectory.asFile + standardOutput = System.out + isIgnoreExitValue = true + } + project.exec { + commandLine("npm", "install") + workingDir = localWebUIRepositoryDirectory.asFile + standardOutput = System.out + } + project.exec { + commandLine("npm", "run", "build") + workingDir = localWebUIRepositoryDirectory.asFile + standardOutput = System.out + } + return@doLast + } + Files.createDirectories(webUICloneDirectory.asFile.toPath()) val cloneResult = project.exec { commandLine("git", "clone", webUIRepositoryAddress) @@ -312,6 +352,20 @@ tasks.named("clean") { tasks.named("processResources") { dependsOn("copyWebUI") + doLast { + Files.copy( + versionFile.toPath(), + layout.buildDirectory.file("resources/main/version.txt").get().asFile.toPath(), + StandardCopyOption.REPLACE_EXISTING + ) + if (webUiPackageJson.exists()) { + Files.copy( + webUiPackageJson.toPath(), + layout.buildDirectory.file("resources/main/webui-package.json").get().asFile.toPath(), + StandardCopyOption.REPLACE_EXISTING + ) + } + } } tasks.named("build") { diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar index 943f0cb..d64cd49 100644 Binary files a/gradle/wrapper/gradle-wrapper.jar and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties index f398c33..1af9e09 100644 --- a/gradle/wrapper/gradle-wrapper.properties +++ b/gradle/wrapper/gradle-wrapper.properties @@ -1,6 +1,7 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-7.6-bin.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-8.5-bin.zip networkTimeout=10000 +validateDistributionUrl=true zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists diff --git a/gradlew b/gradlew index 65dcd68..1aa94a4 100755 --- a/gradlew +++ b/gradlew @@ -83,10 +83,8 @@ done # This is normally unused # shellcheck disable=SC2034 APP_BASE_NAME=${0##*/} -APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit - -# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. -DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' +# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036) +APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit # Use the maximum available, or set MAX_FD != -1 to use that value. MAX_FD=maximum @@ -133,10 +131,13 @@ location of your Java installation." fi else JAVACMD=java - which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + if ! command -v java >/dev/null 2>&1 + then + die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. Please set the JAVA_HOME variable in your environment to match the location of your Java installation." + fi fi # Increase the maximum file descriptors if we can. @@ -144,7 +145,7 @@ if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then case $MAX_FD in #( max*) # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked. - # shellcheck disable=SC3045 + # shellcheck disable=SC2039,SC3045 MAX_FD=$( ulimit -H -n ) || warn "Could not query maximum file descriptor limit" esac @@ -152,7 +153,7 @@ if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then '' | soft) :;; #( *) # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked. - # shellcheck disable=SC3045 + # shellcheck disable=SC2039,SC3045 ulimit -n "$MAX_FD" || warn "Could not set maximum file descriptor limit to $MAX_FD" esac @@ -197,11 +198,15 @@ if "$cygwin" || "$msys" ; then done fi -# Collect all arguments for the java command; -# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of -# shell script including quotes and variable substitutions, so put them in -# double quotes to make sure that they get re-expanded; and -# * put everything else in single quotes, so that it's not re-expanded. + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Collect all arguments for the java command: +# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments, +# and any embedded shellness will be escaped. +# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be +# treated as '${Hostname}' itself on the command line. set -- \ "-Dorg.gradle.appname=$APP_BASE_NAME" \ diff --git a/ldbg.sh b/ldbg.sh new file mode 100755 index 0000000..713b2fc --- /dev/null +++ b/ldbg.sh @@ -0,0 +1,4 @@ +#!/bin/bash +export VERTXWEB_ENVIRONMENT="dev" +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) +VXPORT=5000 DATA_DIR="/mnt/Models/HiCT/data/" TILE_SIZE=256 java -jar ${SCRIPT_DIR}/build/libs/hict_server-*-fat.jar diff --git a/settings.gradle.kts b/settings.gradle.kts index a8452d8..a8f7688 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/AGPProcessor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/AGPProcessor.java index 0ba3025..083e00a 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/AGPProcessor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/AGPProcessor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -179,7 +179,7 @@ yield new GapAGPRecord( } public void initializeContigTreeFromAGP(final @NotNull List<@NotNull AGPFileRecord> agpFileRecords) { - final var originalDescriptors = this.chunkedFile.getOriginalDescriptors(); + final var chunkedFile = this.chunkedFile; final var tree = this.chunkedFile.getContigTree(); final var lock = tree.getRootLock(); try { @@ -189,11 +189,7 @@ public void initializeContigTreeFromAGP(final @NotNull List<@NotNull AGPFileReco if (!(rec instanceof ContigAGPRecord ctgRecord)) { continue; } - final var sourceDescriptor = originalDescriptors.get(ctgRecord.getContigName()); - if (sourceDescriptor == null) { - log.error("Cannot find contig with name " + ctgRecord.getContigName() + " in original .hict.hdf5 file"); - throw new NoSuchElementException("Cannot find contig with name " + ctgRecord.getContigName() + " in original .hict.hdf5 file"); - } + final var sourceDescriptor = chunkedFile.resolveContigDescriptorByName(ctgRecord.getContigName()); final var componentLength = ctgRecord.getInterScaffoldEndIncl() - ctgRecord.getInterScaffoldStartIncl() + 1; if (componentLength != ctgRecord.getIntraContigEndBpIncl() - ctgRecord.getIntraContigStartBpIncl() + 1) { @@ -294,9 +290,9 @@ public Stream getAGPStream(final long unscaffoldedSpacerLength) { for (final var sc : scaffoldedContigs) { final var scaffold = sc.scaffoldTuple().scaffoldDescriptor(); assert (scaffold != null || (sc.contigs().size() == 1)) : "Unscaffolded contig must always represent unique unscaffolded segment"; - final var scaffoldName = (scaffold != null) ? scaffold.scaffoldName() : String.format( + final var scaffoldName = (scaffold != null) ? chunkedFile.getScaffoldDisplayName(scaffold.scaffoldId()) : String.format( "unscaffolded_%s", - sc.contigs().get(0).descriptor().getContigName() + chunkedFile.getContigDisplayName(sc.contigs().get(0).descriptor().getContigId()) ); final var spacerLength = ((scaffold != null) ? scaffold.spacerLength() : unscaffoldedSpacerLength); @@ -305,12 +301,13 @@ public Stream getAGPStream(final long unscaffoldedSpacerLength) { int partNumber = 1; for (final ContigTree.ContigTuple contigTuple : sc.contigs()) { + final var contigDisplayName = chunkedFile.getContigDisplayName(contigTuple.descriptor().getContigId()); result.add(new ContigAGPRecord( scaffoldName, positionBp, positionBp + contigTuple.descriptor().getLengthBp() - 1, partNumber, - contigTuple.descriptor().getContigNameInSourceFASTA(), + contigDisplayName, 1 + contigTuple.descriptor().getOffsetInSourceFASTA(), contigTuple.descriptor().getOffsetInSourceFASTA() + contigTuple.descriptor().getLengthBp(), switch (contigTuple.direction()) { diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/FASTAProcessor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/FASTAProcessor.java index d461c09..bb8f6fe 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/FASTAProcessor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/assembly/FASTAProcessor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ChunkedFile.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ChunkedFile.java index ef85392..0e44915 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ChunkedFile.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ChunkedFile.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -47,14 +47,31 @@ import java.io.IOException; import java.io.Reader; import java.nio.file.Path; +import java.net.URLDecoder; +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; import java.util.Arrays; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.atomic.AtomicInteger; import java.util.stream.LongStream; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBasisATUDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockColsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockLengthDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockOffsetDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockRowsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockValuesDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigHideTypeDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigLengthBinsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigsATLDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getDenseBlockDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getStripeBinWeightsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getStripeLengthsBinsDatasetPath; + @Getter @Slf4j public class ChunkedFile implements AutoCloseable { @@ -74,6 +91,9 @@ public class ChunkedFile implements AutoCloseable { private final @NotNull List> datasetBundlePools; private final @NotNull AGPProcessor agpProcessor; private final @NotNull Map originalDescriptors; + private final @NotNull Map contigNameOverrides = new ConcurrentHashMap<>(); + private final @NotNull Map scaffoldNameOverrides = new ConcurrentHashMap<>(); + private final @NotNull Object nameOverrideLock = new Object(); private final @NotNull TileVisualizationProcessor tileVisualizationProcessor; private final @NotNull FASTAProcessor fastaProcessor; @Getter @@ -85,20 +105,33 @@ public ChunkedFile(final @NotNull ChunkedFileOptions options) { try (final var reader = HDF5Factory.openForReading(this.hdfFilePath.toFile())) { - this.resolutions = LongStream.concat(LongStream.of(0L), reader.object().getAllGroupMembers("/resolutions").parallelStream().filter(s -> { + final var parsedResolutions = reader.object().getAllGroupMembers("/resolutions").parallelStream().flatMap(s -> { try { log.debug("Trying to parse " + s + " as a resolution"); - Long.parseLong(s); + final var parsed = Long.parseLong(s); log.debug("Found new resolution: " + s); - return true; + return java.util.stream.Stream.of(parsed); } catch (final NumberFormatException nfe) { log.debug("Not a resolution: " + s); - return false; + return java.util.stream.Stream.empty(); } - }).mapToLong(Long::parseLong)).sorted().toArray(); + }).sorted().toList(); + + final var validResolutions = parsedResolutions.stream() + .filter(resolution -> isResolutionComplete(reader, resolution)) + .sorted() + .toList(); + + if (validResolutions.isEmpty()) { + throw new IllegalStateException("No complete resolutions found in " + this.hdfFilePath); + } + this.resolutions = LongStream.concat(LongStream.of(0L), validResolutions.stream().mapToLong(Long::longValue)).sorted().toArray(); - this.denseBlockSize = (int) Arrays.stream(resolutions).sorted().skip(1L).map(res -> reader.int64().getAttr(String.format("/resolutions/%d/treap_coo", res), "dense_submatrix_size")).max().orElse(256L); + this.denseBlockSize = (int) validResolutions.stream() + .mapToLong(res -> reader.int64().getAttr(String.format("/resolutions/%d/treap_coo", res), "dense_submatrix_size")) + .max() + .orElse(256L); log.info("Dense block size: " + this.denseBlockSize); log.debug("Resolutions count: " + resolutions.length); @@ -144,6 +177,7 @@ public ChunkedFile(final @NotNull ChunkedFileOptions options) { this.agpProcessor = new AGPProcessor(this); this.tileVisualizationProcessor = new TileVisualizationProcessor(this); this.fastaProcessor = new FASTAProcessor(this); + this.loadNameOverrides(); this.resolutionScalingCoefficient = new double[this.resolutions.length]; this.resolutionLinearScalingCoefficient = new double[this.resolutions.length]; @@ -158,6 +192,151 @@ public ChunkedFile(final @NotNull ChunkedFileOptions options) { } } + private void loadNameOverrides() { + // Name overrides are session-only and should not be loaded from the HDF5 file. + } + + private static void readOverrideMap(final @NotNull String encoded, final @NotNull Map target, final @NotNull java.util.function.Function keyParser) { + if (encoded.isBlank()) { + return; + } + for (final var line : encoded.split("\n")) { + if (line.isBlank()) { + continue; + } + final var parts = line.split("\t", 2); + if (parts.length != 2) { + continue; + } + final var key = keyParser.apply(parts[0]); + final var value = URLDecoder.decode(parts[1], StandardCharsets.UTF_8); + if (!value.isBlank()) { + target.put(key, value); + } + } + } + + private static @NotNull String writeOverrideMap(final @NotNull Map source) { + final var sb = new StringBuilder(); + for (final var entry : source.entrySet()) { + final var name = entry.getValue(); + if (name == null || name.isBlank()) { + continue; + } + if (!sb.isEmpty()) { + sb.append('\n'); + } + sb.append(entry.getKey()).append('\t').append(URLEncoder.encode(name, StandardCharsets.UTF_8)); + } + return sb.toString(); + } + + private void persistNameOverrides() { + // Name overrides are session-only and should not be persisted to the HDF5 file. + } + + public @NotNull String getContigOriginalName(final int contigId) { + final var descriptor = this.contigTree.getContigDescriptors().get(contigId); + if (descriptor == null) { + throw new IllegalArgumentException("Unknown contig id " + contigId); + } + return descriptor.getContigName(); + } + + public @NotNull String getContigDisplayName(final int contigId) { + return Optional.ofNullable(contigNameOverrides.get(contigId)).orElse(getContigOriginalName(contigId)); + } + + public @NotNull String getScaffoldOriginalName(final long scaffoldId) { + return this.scaffoldTree.getScaffoldList().stream() + .filter(tuple -> tuple.scaffoldDescriptor().scaffoldId() == scaffoldId) + .findFirst() + .map(tuple -> tuple.scaffoldDescriptor().scaffoldName()) + .orElseThrow(() -> new IllegalArgumentException("Unknown scaffold id " + scaffoldId)); + } + + public @NotNull String getScaffoldDisplayName(final long scaffoldId) { + return Optional.ofNullable(scaffoldNameOverrides.get(scaffoldId)).orElse(getScaffoldOriginalName(scaffoldId)); + } + + public void setContigNameOverride(final int contigId, final @NotNull String newName) { + if (newName.isBlank() || newName.equals(getContigOriginalName(contigId))) { + contigNameOverrides.remove(contigId); + } else { + contigNameOverrides.put(contigId, newName); + } + persistNameOverrides(); + } + + public void setScaffoldNameOverride(final long scaffoldId, final @NotNull String newName) { + if (newName.isBlank() || newName.equals(getScaffoldOriginalName(scaffoldId))) { + scaffoldNameOverrides.remove(scaffoldId); + } else { + scaffoldNameOverrides.put(scaffoldId, newName); + } + persistNameOverrides(); + } + + public @NotNull Map getContigNameOverrides() { + return contigNameOverrides; + } + + public @NotNull Map getScaffoldNameOverrides() { + return scaffoldNameOverrides; + } + + public @NotNull java.util.Set getAllContigDisplayNames() { + return this.contigTree.getContigDescriptors().keySet().stream().map(this::getContigDisplayName).collect(java.util.stream.Collectors.toSet()); + } + + public @NotNull java.util.Set getAllScaffoldDisplayNames() { + return this.scaffoldTree.getScaffoldList().stream() + .map(tuple -> Optional.ofNullable(scaffoldNameOverrides.get(tuple.scaffoldDescriptor().scaffoldId())).orElse(tuple.scaffoldDescriptor().scaffoldName())) + .collect(java.util.stream.Collectors.toSet()); + } + + public @NotNull ContigDescriptor resolveContigDescriptorByName(final @NotNull String contigName) { + final var original = originalDescriptors.get(contigName); + if (original != null) { + return original; + } + for (final var entry : contigNameOverrides.entrySet()) { + if (contigName.equals(entry.getValue())) { + final var descriptor = this.contigTree.getContigDescriptors().get(entry.getKey()); + if (descriptor != null) { + return descriptor; + } + } + } + throw new IllegalArgumentException("Unknown contig name " + contigName); + } + + private static boolean isResolutionComplete(final @NotNull ch.systemsx.cisd.hdf5.IHDF5Reader reader, final long resolution) { + final String base = "/resolutions/" + resolution; + try { + if (!reader.object().isGroup(base + "/treap_coo")) { + log.warn("Skipping resolution {}: missing treap_coo group", resolution); + return false; + } + if (!reader.object().isDataSet(getBlockLengthDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getBlockOffsetDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getBlockRowsDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getBlockColsDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getBlockValuesDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getDenseBlockDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getStripeLengthsBinsDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getStripeBinWeightsDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getContigLengthBinsDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getContigHideTypeDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getContigsATLDatasetPath(resolution))) return false; + if (!reader.object().isDataSet(getBasisATUDatasetPath(resolution))) return false; + return true; + } catch (Exception e) { + log.warn("Skipping resolution {} due to validation error: {}", resolution, e.getMessage()); + return false; + } + } + public @NotNull MatrixQueries matrixQueries() { return this.matrixQueries; } diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/Initializers.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/Initializers.java index 4cf96db..1f93bd5 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/Initializers.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/Initializers.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/MatrixQueries.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/MatrixQueries.java index 18a30a1..9d0d546 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/MatrixQueries.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/MatrixQueries.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ScaffoldingOperations.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ScaffoldingOperations.java index a289ca3..4e533de 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ScaffoldingOperations.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/ScaffoldingOperations.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -158,7 +158,7 @@ public void unscaffoldRegion(final long startIncl, final long endExcl, final @No final var extended = scaffoldTree.extendBordersToScaffolds(lessSize, lessSize + segmentSize); - scaffoldTree.removeSegmentFromAssembly(extended.startBP(), extended.endBP()); + scaffoldTree.unscaffold(extended.startBP(), extended.endBP()); } finally { scaffoldTree.getRootLock().writeLock().unlock(); diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundle.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundle.java index bd23e80..1016221 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundle.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundle.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundleFactory.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundleFactory.java index b8eb3fd..c9335b4 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundleFactory.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5FileDatasetsBundleFactory.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5LibraryInitializer.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5LibraryInitializer.java index 584141f..625e143 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5LibraryInitializer.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/hdf5/HDF5LibraryInitializer.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/resolution/ResolutionDescriptor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/resolution/ResolutionDescriptor.java index cc996cf..2f3a9d3 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/resolution/ResolutionDescriptor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/resolution/ResolutionDescriptor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/util/PathGenerators.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/util/PathGenerators.java index cb157d3..26e37ae 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/util/PathGenerators.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/chunkedfile/util/PathGenerators.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/ConversionOptions.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/ConversionOptions.java new file mode 100644 index 0000000..ba9d740 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/ConversionOptions.java @@ -0,0 +1,54 @@ +package ru.itmo.ctlab.hict.hict_library.converters; + +import org.jetbrains.annotations.NotNull; + +import java.nio.file.Path; +import java.util.List; + +public record ConversionOptions( + @NotNull Path inputPath, + @NotNull Path outputPath, + @NotNull List<@NotNull Long> resolutions, + int chunkSize, + int compressionLevel, + @NotNull CompressionAlgorithm compressionAlgorithm, + @NotNull String agpPath, + boolean applyAgpBeforeExport, + int parallelism +) { + public static final String NO_AGP = ""; + + public ConversionOptions { + if (chunkSize <= 0) { + chunkSize = 8_192; + } + if (compressionLevel < 0 || compressionLevel > 9) { + compressionLevel = 0; + } + if (compressionAlgorithm == null) { + compressionAlgorithm = CompressionAlgorithm.DEFLATE; + } + if (agpPath == null) { + agpPath = NO_AGP; + } + if (parallelism <= 0) { + parallelism = Math.max(1, Runtime.getRuntime().availableProcessors()); + } + } + + public enum CompressionAlgorithm { + DEFLATE, + ZSTD, + LZF; + + public static @NotNull CompressionAlgorithm parse(final @NotNull String value) { + final var normalized = value.trim().toUpperCase(); + return switch (normalized) { + case "DEFLATE" -> DEFLATE; + case "ZSTD" -> ZSTD; + case "LZF" -> LZF; + default -> throw new IllegalArgumentException("Unknown compression algorithm: " + value + " (expected: deflate|zstd|lzf)"); + }; + } + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/HictToMcoolConverter.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/HictToMcoolConverter.java new file mode 100644 index 0000000..37844cc --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/HictToMcoolConverter.java @@ -0,0 +1,279 @@ +package ru.itmo.ctlab.hict.hict_library.converters; + +import ch.systemsx.cisd.hdf5.HDF5Factory; +import ch.systemsx.cisd.hdf5.HDF5IntStorageFeatures; +import io.vertx.core.json.JsonArray; +import org.jetbrains.annotations.NotNull; +import ru.itmo.ctlab.hict.hict_library.chunkedfile.ChunkedFile; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.concurrent.*; +import java.util.function.Consumer; + +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.*; + +public class HictToMcoolConverter { + + public void convert(final @NotNull ConversionOptions options, final @NotNull Consumer logConsumer) throws IOException, NoSuchFieldException { + final var synchronizedLogConsumer = synchronizedLogger(logConsumer); + final var chunkedFile = new ChunkedFile(new ChunkedFile.ChunkedFileOptions(options.inputPath(), 2, 8)); + try { + if (options.applyAgpBeforeExport() && !options.agpPath().isBlank()) { + try (final var reader = Files.newBufferedReader(options.inputPath().resolveSibling(options.agpPath()), StandardCharsets.UTF_8)) { + chunkedFile.importAGP(reader); + } + synchronizedLogConsumer.accept("Applied AGP before export"); + } + + final var selectedResolutions = resolveResolutions(chunkedFile.getResolutions(), options.resolutions()); + final var compression = resolveIntStorageFeatures(options, synchronizedLogConsumer); + final var requestedWorkers = resolveRequestedWorkers(options.parallelism()); + final var workers = Math.max(1, Math.min(requestedWorkers, selectedResolutions.size())); + + synchronizedLogConsumer.accept("Converting in parallel with workers=" + workers + ", chunkSize=" + options.chunkSize()); + + final var stagedResolutionFiles = convertResolutionsInParallel( + options.inputPath(), + selectedResolutions, + options.chunkSize(), + compression, + workers, + synchronizedLogConsumer + ); + + try (final var dst = HDF5Factory.open(options.outputPath().toFile())) { + dst.object().createGroup("/resolutions"); + dst.string().write("/format", "mcool-lite"); + dst.string().write("/source_format", "hict"); + dst.string().write("/selected_resolutions", new JsonArray(selectedResolutions).encode()); + + for (final var staged : stagedResolutionFiles.stream().sorted(Comparator.comparingLong(StagedResolutionFile::resolution)).toList()) { + try (final var stagedReader = HDF5Factory.openForReading(staged.path().toFile())) { + mergeResolution(stagedReader, dst, staged.resolution(), options.chunkSize(), compression, logConsumer); + synchronizedLogConsumer.accept("Merged resolution " + staged.resolution() + " to final output"); + } finally { + try { + Files.deleteIfExists(staged.path()); + } catch (IOException e) { + synchronizedLogConsumer.accept("Failed to delete temp file " + staged.path() + ": " + e.getMessage()); + } + } + } + } + } finally { + chunkedFile.close(); + } + } + + private static @NotNull List convertResolutionsInParallel( + final @NotNull Path inputPath, + final @NotNull List selectedResolutions, + final int chunkSize, + final @NotNull HDF5IntStorageFeatures compression, + final int workers, + final @NotNull Consumer logConsumer + ) { + final ExecutorService executor = Executors.newFixedThreadPool(workers); + final List> futures = new ArrayList<>(); + + for (final var resolution : selectedResolutions) { + futures.add(executor.submit(() -> { + final var stagedFile = Files.createTempFile("hict-to-mcool-r" + resolution + "-", ".h5"); + try (final var src = HDF5Factory.openForReading(inputPath.toFile()); + final var dst = HDF5Factory.open(stagedFile.toFile())) { + stageResolution(src, dst, resolution, chunkSize, compression, logConsumer); + return new StagedResolutionFile(resolution, stagedFile); + } catch (Exception e) { + Files.deleteIfExists(stagedFile); + throw e; + } + })); + } + + final var out = new ArrayList(); + try { + for (final var f : futures) { + out.add(f.get()); + } + return out; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } finally { + executor.shutdownNow(); + } + } + + private static void stageResolution( + final @NotNull ch.systemsx.cisd.hdf5.IHDF5Reader src, + final @NotNull ch.systemsx.cisd.hdf5.IHDF5Writer dst, + final long resolution, + final int chunkSize, + final @NotNull HDF5IntStorageFeatures compression, + final @NotNull Consumer logConsumer + ) { + final var root = "/resolutions/" + resolution; + dst.object().createGroup("/resolutions"); + dst.object().createGroup(root); + dst.object().createGroup(root + "/pixels"); + dst.object().createGroup(root + "/indexes"); + + final var prefix = "Resolution " + resolution; + copyLongArrayChunked(src, dst, getBlockLengthDatasetPath(resolution), root + "/pixels/count", chunkSize, compression, logConsumer, prefix + " pixels/count"); + copyLongArrayChunked(src, dst, getBlockOffsetDatasetPath(resolution), root + "/indexes/block_offset", chunkSize, compression, logConsumer, prefix + " indexes/block_offset"); + copyLongArrayChunked(src, dst, getBlockRowsDatasetPath(resolution), root + "/pixels/bin1_id", chunkSize, compression, logConsumer, prefix + " pixels/bin1_id"); + copyLongArrayChunked(src, dst, getBlockColsDatasetPath(resolution), root + "/pixels/bin2_id", chunkSize, compression, logConsumer, prefix + " pixels/bin2_id"); + copyLongArrayChunked(src, dst, getBlockValuesDatasetPath(resolution), root + "/pixels/counts", chunkSize, compression, logConsumer, prefix + " pixels/counts"); + copyLongArrayChunked(src, dst, getDenseBlockDatasetPath(resolution), root + "/pixels/dense_blocks", chunkSize, compression, logConsumer, prefix + " pixels/dense_blocks"); + dst.int64().setAttr(root, "bin_size", resolution); + logConsumer.accept("Staged resolution " + resolution + " in worker=" + Thread.currentThread().getName()); + } + + private static void mergeResolution( + final @NotNull ch.systemsx.cisd.hdf5.IHDF5Reader stagedReader, + final @NotNull ch.systemsx.cisd.hdf5.IHDF5Writer dst, + final long resolution, + final int chunkSize, + final @NotNull HDF5IntStorageFeatures compression, + final @NotNull Consumer logConsumer + ) { + final var root = "/resolutions/" + resolution; + dst.object().createGroup(root); + dst.object().createGroup(root + "/pixels"); + dst.object().createGroup(root + "/indexes"); + + final var prefix = "Merge resolution " + resolution; + copyLongArrayChunked(stagedReader, dst, root + "/pixels/count", root + "/pixels/count", chunkSize, compression, logConsumer, prefix + " pixels/count"); + copyLongArrayChunked(stagedReader, dst, root + "/indexes/block_offset", root + "/indexes/block_offset", chunkSize, compression, logConsumer, prefix + " indexes/block_offset"); + copyLongArrayChunked(stagedReader, dst, root + "/pixels/bin1_id", root + "/pixels/bin1_id", chunkSize, compression, logConsumer, prefix + " pixels/bin1_id"); + copyLongArrayChunked(stagedReader, dst, root + "/pixels/bin2_id", root + "/pixels/bin2_id", chunkSize, compression, logConsumer, prefix + " pixels/bin2_id"); + copyLongArrayChunked(stagedReader, dst, root + "/pixels/counts", root + "/pixels/counts", chunkSize, compression, logConsumer, prefix + " pixels/counts"); + copyLongArrayChunked(stagedReader, dst, root + "/pixels/dense_blocks", root + "/pixels/dense_blocks", chunkSize, compression, logConsumer, prefix + " pixels/dense_blocks"); + dst.int64().setAttr(root, "bin_size", resolution); + } + + static void copyLongArrayChunked( + final @NotNull ch.systemsx.cisd.hdf5.IHDF5Reader src, + final @NotNull ch.systemsx.cisd.hdf5.IHDF5Writer dst, + final @NotNull String srcPath, + final @NotNull String dstPath, + final int chunkSize, + final @NotNull HDF5IntStorageFeatures compression, + final @NotNull Consumer logConsumer, + final @NotNull String progressLabel + ) { + if (!src.object().isDataSet(srcPath)) { + logConsumer.accept("Skipped missing dataset " + srcPath); + return; + } + + final var dims = src.object().getDataSetInformation(srcPath).getDimensions(); + final long length = dims.length == 0 ? 0 : dims[0]; + dst.int64().createArray(dstPath, length, chunkSize, compression); + + long offset = 0L; + final long startedNanos = System.nanoTime(); + int lastLoggedPercent = -1; + while (offset < length) { + final int blockLen = (int) Math.min(chunkSize, length - offset); + final var block = src.int64().readArrayBlockWithOffset(srcPath, blockLen, offset); + dst.int64().writeArrayBlockWithOffset(dstPath, block, blockLen, offset); + offset += blockLen; + if (length > 0) { + final int percent = (int) ((offset * 100L) / length); + if (percent >= 100 || percent - lastLoggedPercent >= 10) { + lastLoggedPercent = percent; + final long elapsedMillis = (System.nanoTime() - startedNanos) / 1_000_000L; + final long etaMillis = estimateEtaMillis(offset, length, elapsedMillis); + logConsumer.accept( + String.format( + "%s: %d%% (%d/%d), elapsed=%s, eta=%s", + progressLabel, + percent, + offset, + length, + formatDuration(elapsedMillis), + formatDuration(etaMillis) + ) + ); + } + } + } + logConsumer.accept("Copied " + srcPath + " -> " + dstPath + " (" + length + " items)"); + } + + private @NotNull List resolveResolutions(final long @NotNull [] availableResolutions, final @NotNull List requested) { + final var available = new ArrayList(); + for (int i = 1; i < availableResolutions.length; i++) { + available.add(availableResolutions[i]); + } + if (requested == null || requested.isEmpty()) { + return available; + } + return available.stream().filter(requested::contains).toList(); + } + + private record StagedResolutionFile(long resolution, @NotNull Path path) { + } + + private static @NotNull HDF5IntStorageFeatures resolveIntStorageFeatures(final @NotNull ConversionOptions options, final @NotNull Consumer logConsumer) { + if (options.compressionLevel() <= 0) { + return HDF5IntStorageFeatures.INT_CHUNKED; + } + return switch (options.compressionAlgorithm()) { + case DEFLATE -> HDF5IntStorageFeatures.createDeflation(options.compressionLevel()); + case ZSTD, LZF -> { + logConsumer.accept( + "Compression algorithm " + options.compressionAlgorithm() + + " requested, but current JHDF5 high-level writer path supports deflate features only. Falling back to Deflate." + ); + yield HDF5IntStorageFeatures.createDeflation(options.compressionLevel()); + } + }; + } + + private static long estimateEtaMillis(final long done, final long total, final long elapsedMillis) { + if (done <= 0 || total <= 0 || done >= total || elapsedMillis <= 0) { + return 0L; + } + return (elapsedMillis * (total - done)) / done; + } + + private static @NotNull String formatDuration(final long millis) { + if (millis <= 0) { + return "00:00"; + } + final long totalSeconds = millis / 1000L; + final long hours = totalSeconds / 3600L; + final long minutes = (totalSeconds % 3600L) / 60L; + final long seconds = totalSeconds % 60L; + if (hours > 0) { + return String.format("%02d:%02d:%02d", hours, minutes, seconds); + } + return String.format("%02d:%02d", minutes, seconds); + } + + private static @NotNull Consumer synchronizedLogger(final @NotNull Consumer delegate) { + final Object lock = new Object(); + return message -> { + synchronized (lock) { + delegate.accept(message); + } + }; + } + + private static int resolveRequestedWorkers(final int parallelismOption) { + if (parallelismOption == -1 || parallelismOption <= 0) { + return Math.max(1, Runtime.getRuntime().availableProcessors()); + } + return parallelismOption; + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/McoolToHictConverter.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/McoolToHictConverter.java new file mode 100644 index 0000000..95bc6a8 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/converters/McoolToHictConverter.java @@ -0,0 +1,1030 @@ +package ru.itmo.ctlab.hict.hict_library.converters; + +import ch.systemsx.cisd.base.mdarray.MDLongArray; +import ch.systemsx.cisd.hdf5.HDF5Factory; +import ch.systemsx.cisd.hdf5.HDF5FloatStorageFeatures; +import ch.systemsx.cisd.hdf5.HDF5IntStorageFeatures; +import ch.systemsx.cisd.hdf5.IHDF5Reader; +import ch.systemsx.cisd.hdf5.IHDF5Writer; +import org.jetbrains.annotations.NotNull; +import ru.itmo.ctlab.hict.hict_library.domain.ATUDescriptor; +import ru.itmo.ctlab.hict.hict_library.domain.ATUDirection; +import ru.itmo.ctlab.hict.hict_library.domain.ContigDirection; +import ru.itmo.ctlab.hict.hict_library.domain.ContigHideType; +import ru.itmo.ctlab.hict.hict_library.domain.StripeDescriptor; + +import java.io.IOException; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReferenceArray; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; + +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBasisATUDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockColsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockLengthDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockOffsetDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockRowsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getBlockValuesDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigDirectionDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigHideTypeDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigLengthBinsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigLengthBpDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigNameDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigOrderDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getContigsATLDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getDenseBlockDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getStripeBinWeightsDatasetPath; +import static ru.itmo.ctlab.hict.hict_library.chunkedfile.util.PathGenerators.getStripeLengthsBinsDatasetPath; + +public class McoolToHictConverter { + private static final int SUBMATRIX_SIZE = 256; + private static final long HDF5_MAX_CHUNK_SIZE = 32L * 1024L * 1024L * 8L; + private static final int DENSE_THRESHOLD = (SUBMATRIX_SIZE * SUBMATRIX_SIZE) / 2; + + public void convert(final @NotNull ConversionOptions options, final @NotNull Consumer logConsumer) { + final var synchronizedLogConsumer = synchronizedLogger(logConsumer); + try (final var src = HDF5Factory.openForReading(options.inputPath().toFile())) { + final var selectedResolutions = resolveResolutions(src, options.resolutions()); + if (selectedResolutions.isEmpty()) { + throw new IllegalArgumentException("No numeric resolutions found in input file"); + } + + final var conversionOrder = selectedResolutions.stream().sorted(Comparator.reverseOrder()).toList(); + final var intStorageFeatures = resolveIntStorageFeatures(options, synchronizedLogConsumer); + final var floatStorageFeatures = resolveFloatStorageFeatures(options, synchronizedLogConsumer); + final var progressTracker = new ConversionProgressTracker((conversionOrder.size() * 2) + 1, synchronizedLogConsumer); + + final var requestedWorkers = resolveRequestedWorkers(options.parallelism()); + synchronizedLogConsumer.accept( + "Converting .mcool -> .hict.hdf5, workers=" + requestedWorkers + ", resolutions=" + conversionOrder + + ", compressionAlgorithm=" + options.compressionAlgorithm() + ", compressionLevel=" + options.compressionLevel() + ); + + Files.deleteIfExists(options.outputPath()); + try (final var dst = HDF5Factory.open(options.outputPath().toFile()); + final var srcAgain = HDF5Factory.openForReading(options.inputPath().toFile())) { + dst.object().createGroup("/resolutions"); + dst.string().setAttr("/resolutions", "hict_version", "0.1.3.1a"); + + dumpContigData(srcAgain, dst, selectedResolutions, requestedWorkers, intStorageFeatures, floatStorageFeatures); + progressTracker.markStep("Dumped contig metadata"); + + for (final var resolution : conversionOrder) { + writeResolutionDirect( + srcAgain, + dst, + resolution, + options.chunkSize(), + intStorageFeatures, + floatStorageFeatures, + requestedWorkers, + synchronizedLogConsumer + ); + progressTracker.markStep("Wrote resolution " + resolution); + } + } + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + private static @NotNull HDF5IntStorageFeatures resolveIntStorageFeatures(final @NotNull ConversionOptions options, final @NotNull Consumer logConsumer) { + if (options.compressionLevel() <= 0) { + return HDF5IntStorageFeatures.INT_CHUNKED; + } + return switch (options.compressionAlgorithm()) { + case DEFLATE -> HDF5IntStorageFeatures.createDeflation(options.compressionLevel()); + case ZSTD, LZF -> { + logConsumer.accept( + "Compression algorithm " + options.compressionAlgorithm() + + " requested, but current JHDF5 high-level writer path supports deflate features only. Falling back to uncompressed chunked datasets." + ); + yield HDF5IntStorageFeatures.INT_CHUNKED; + } + }; + } + + private static @NotNull HDF5FloatStorageFeatures resolveFloatStorageFeatures(final @NotNull ConversionOptions options, final @NotNull Consumer logConsumer) { + if (options.compressionLevel() <= 0) { + return HDF5FloatStorageFeatures.FLOAT_CHUNKED; + } + return switch (options.compressionAlgorithm()) { + case DEFLATE -> HDF5FloatStorageFeatures.createDeflation(options.compressionLevel()); + case ZSTD, LZF -> { + logConsumer.accept( + "Compression algorithm " + options.compressionAlgorithm() + + " requested, but current JHDF5 high-level writer path supports deflate features only. Falling back to uncompressed chunked datasets." + ); + yield HDF5FloatStorageFeatures.FLOAT_CHUNKED; + } + }; + } + + private static void writeResolutionDirect( + final @NotNull IHDF5Reader src, + final @NotNull IHDF5Writer dst, + final long resolution, + final int chunkSize, + final @NotNull HDF5IntStorageFeatures intStorageFeatures, + final @NotNull HDF5FloatStorageFeatures floatStorageFeatures, + final int stripeWorkersRequested, + final @NotNull Consumer logConsumer + ) { + final long startedNanos = System.nanoTime(); + checkInterrupted(); + final var resolutionRoot = "/resolutions/" + resolution; + if (!dst.object().isGroup(resolutionRoot)) { + dst.object().createGroup(resolutionRoot); + } + + final var nameLengthPath = resolveNameLengthPath(src, resolution); + final var stripes = dumpStripeData(src, dst, resolution, nameLengthPath, floatStorageFeatures); + + final var treapRoot = resolutionRoot + "/treap_coo"; + dst.object().createGroup(treapRoot); + dst.int64().setAttr(treapRoot, "dense_submatrix_size", SUBMATRIX_SIZE); + dst.int64().setAttr(treapRoot, "hdf5_max_chunk_size", HDF5_MAX_CHUNK_SIZE); + + final var binsCount = datasetLength(src, resolutionRoot + "/bins/end"); + final var nonzeroPixelCount = datasetLength(src, resolutionRoot + "/pixels/bin1_id"); + final var stripeCount = stripes.size(); + dst.int64().setAttr(treapRoot, "bins_count", binsCount); + dst.int64().setAttr(treapRoot, "stripes_count", stripeCount); + + final var allRowsStartIndices = src.int64().readArray(resolutionRoot + "/indexes/bin1_offset"); + + logConsumer.accept("Resolution " + resolution + ": counting sparse and dense blocks"); + final var countingProgress = new PhaseProgressTracker( + "Resolution " + resolution + " count", + stripeCount, + logConsumer + ); + final int stripeWorkers = Math.max(1, Math.min(stripeWorkersRequested, Math.max(1, stripeCount))); + final var counts = countDenseAndSparse(src, resolution, stripeCount, allRowsStartIndices, stripeWorkers, countingProgress::report); + countingProgress.finish(); + final var denseBlockCount = counts.denseBlockCount(); + logConsumer.accept("Resolution " + resolution + ": finished counting blocks, denseBlocks=" + denseBlockCount); + + final var blockRowsPath = getBlockRowsDatasetPath(resolution); + final var blockColsPath = getBlockColsDatasetPath(resolution); + final var blockValsPath = getBlockValuesDatasetPath(resolution); + final var blockOffsetPath = getBlockOffsetDatasetPath(resolution); + final var blockLengthPath = getBlockLengthDatasetPath(resolution); + final var denseBlocksPath = getDenseBlockDatasetPath(resolution); + + dst.int64().createArray(blockRowsPath, nonzeroPixelCount, safeChunkLen(nonzeroPixelCount, chunkSize), intStorageFeatures); + dst.int64().createArray(blockColsPath, nonzeroPixelCount, safeChunkLen(nonzeroPixelCount, chunkSize), intStorageFeatures); + dst.int64().createArray(blockValsPath, nonzeroPixelCount, safeChunkLen(nonzeroPixelCount, chunkSize), intStorageFeatures); + + final var totalBlockCount = (long) stripeCount * stripeCount; + dst.int64().createArray(blockOffsetPath, totalBlockCount, safeChunkLen(totalBlockCount, chunkSize), intStorageFeatures); + dst.int64().createArray(blockLengthPath, totalBlockCount, safeChunkLen(totalBlockCount, chunkSize), intStorageFeatures); + + final var denseDatasetSize = Math.max(1L, denseBlockCount); + dst.int64().createMDArray( + denseBlocksPath, + new long[]{denseDatasetSize, 1L, SUBMATRIX_SIZE, SUBMATRIX_SIZE}, + new int[]{1, 1, SUBMATRIX_SIZE, SUBMATRIX_SIZE}, + intStorageFeatures + ); + + long currentSparseOffset = 0L; + long currentDenseOffset = 0L; + final var writeProgress = new PhaseProgressTracker( + "Resolution " + resolution + " write", + stripeCount, + logConsumer + ); + if (stripeCount == 0) { + writeProgress.finish(); + return; + } + + final var sortedStripes = new AtomicReferenceArray(stripeCount); + final var errorRef = new AtomicReference(); + final Object lock = new Object(); + final ExecutorService stripeExecutor = Executors.newFixedThreadPool(stripeWorkers); + final Object readLock = new Object(); + final List> futures = new ArrayList<>(stripeCount); + + for (int rowStripe = 0; rowStripe < stripeCount; rowStripe++) { + checkInterrupted(); + final int stripeIdx = rowStripe; + futures.add(stripeExecutor.submit(() -> { + try { + final PixelBlock block; + synchronized (readLock) { + checkInterrupted(); + block = readRowStripePixels(src, resolution, stripeIdx, allRowsStartIndices); + } + final SortedStripePixels sorted = block.length() > 0 + ? sortStripePixels(block.rows(), block.cols(), block.values()) + : EMPTY_STRIPE; + sortedStripes.set(stripeIdx, sorted); + } catch (Throwable t) { + errorRef.compareAndSet(null, t); + } finally { + synchronized (lock) { + lock.notifyAll(); + } + } + })); + } + + try { + for (int rowStripe = 0; rowStripe < stripeCount; rowStripe++) { + checkInterrupted(); + SortedStripePixels sorted = sortedStripes.get(rowStripe); + while (sorted == null) { + if (errorRef.get() != null) { + throw new RuntimeException(errorRef.get()); + } + synchronized (lock) { + try { + lock.wait(50L); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } + } + sorted = sortedStripes.get(rowStripe); + } + + if (sorted != EMPTY_STRIPE) { + final var saveResult = saveIndirectBlock( + dst, + rowStripe, + stripeCount, + sorted, + currentSparseOffset, + currentDenseOffset, + blockRowsPath, + blockColsPath, + blockValsPath, + blockOffsetPath, + blockLengthPath, + denseBlocksPath + ); + currentSparseOffset = saveResult.sparseOffset(); + currentDenseOffset = saveResult.denseOffset(); + } + sortedStripes.set(rowStripe, null); + writeProgress.report(rowStripe + 1); + + final int percent = (int) ((rowStripe + 1L) * 100L / stripeCount); + final var elapsedMillis = (System.nanoTime() - startedNanos) / 1_000_000L; + final var etaMillis = estimateEtaMillis(rowStripe + 1L, stripeCount, elapsedMillis); + logConsumer.accept( + String.format( + "Resolution %d write: %d%% (%d/%d stripes), elapsed=%s, eta=%s", + resolution, + percent, + rowStripe + 1, + stripeCount, + formatDuration(elapsedMillis), + formatDuration(etaMillis) + ) + ); + } + } finally { + stripeExecutor.shutdown(); + } + + try { + for (final var f : futures) { + f.get(); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } + writeProgress.finish(); + } + + private static @NotNull Consumer synchronizedLogger(final @NotNull Consumer delegate) { + final Object lock = new Object(); + return msg -> { + synchronized (lock) { + delegate.accept(msg); + } + }; + } + + private static SaveBlockResult saveIndirectBlock( + final @NotNull IHDF5Writer dst, + final int rowStripe, + final int stripeCount, + final @NotNull SortedStripePixels sorted, + final long currentSparseOffset, + final long currentDenseOffset, + final @NotNull String blockRowsPath, + final @NotNull String blockColsPath, + final @NotNull String blockValsPath, + final @NotNull String blockOffsetPath, + final @NotNull String blockLengthPath, + final @NotNull String denseBlocksPath + ) { + long sparseOffset = currentSparseOffset; + long denseOffset = currentDenseOffset; + + final var colStripes = sorted.colStripes(); + final var intraRows = sorted.intraRows(); + final var intraCols = sorted.intraCols(); + final var values = sorted.values(); + + int start = 0; + while (start < colStripes.length) { + int end = start + 1; + final long colStripe = colStripes[start]; + while (end < colStripes.length && colStripes[end] == colStripe) { + end++; + } + + final int blockLen = end - start; + if (blockLen > 0) { + final long blockIndex = (long) rowStripe * stripeCount + colStripe; + + if (blockLen >= DENSE_THRESHOLD) { + dst.int64().writeArrayBlockWithOffset(blockOffsetPath, new long[]{-denseOffset - 1L}, 1, blockIndex); + dst.int64().writeArrayBlockWithOffset(blockLengthPath, new long[]{blockLen}, 1, blockIndex); + + final var denseFlat = new long[SUBMATRIX_SIZE * SUBMATRIX_SIZE]; + for (int i = start; i < end; i++) { + final int r = intraRows[i]; + final int c = intraCols[i]; + denseFlat[r * SUBMATRIX_SIZE + c] += values[i]; + } + + final var denseMd = new MDLongArray(denseFlat, new int[]{1, 1, SUBMATRIX_SIZE, SUBMATRIX_SIZE}); + dst.int64().writeMDArrayBlockWithOffset(denseBlocksPath, denseMd, new long[]{denseOffset, 0L, 0L, 0L}); + denseOffset++; + } else { + dst.int64().writeArrayBlockWithOffset(blockOffsetPath, new long[]{sparseOffset}, 1, blockIndex); + dst.int64().writeArrayBlockWithOffset(blockLengthPath, new long[]{blockLen}, 1, blockIndex); + + final var blockRows = new long[blockLen]; + final var blockCols = new long[blockLen]; + final var blockVals = new long[blockLen]; + for (int i = 0; i < blockLen; i++) { + blockRows[i] = intraRows[start + i]; + blockCols[i] = intraCols[start + i]; + blockVals[i] = values[start + i]; + } + + dst.int64().writeArrayBlockWithOffset(blockRowsPath, blockRows, blockLen, sparseOffset); + dst.int64().writeArrayBlockWithOffset(blockColsPath, blockCols, blockLen, sparseOffset); + dst.int64().writeArrayBlockWithOffset(blockValsPath, blockVals, blockLen, sparseOffset); + + sparseOffset += blockLen; + } + } + + start = end; + } + + return new SaveBlockResult(sparseOffset, denseOffset); + } + + private static @NotNull StripeCounts countDenseAndSparse( + final @NotNull IHDF5Reader src, + final long resolution, + final int stripeCount, + final long @NotNull [] allRowsStartIndices, + final int stripeWorkers, + final @NotNull java.util.function.IntConsumer countingProgressReporter + ) { + long sparseCount = 0L; + long denseCount = 0L; + + final int batchSize = Math.max(1, stripeWorkers * 2); + final ExecutorService stripeExecutor = stripeWorkers > 1 ? Executors.newFixedThreadPool(stripeWorkers) : null; + try { + for (int batchStart = 0; batchStart < stripeCount; batchStart += batchSize) { + checkInterrupted(); + final int batchEnd = Math.min(stripeCount, batchStart + batchSize); + final int localCount = batchEnd - batchStart; + final var blocks = new PixelBlock[localCount]; + for (int i = 0; i < localCount; i++) { + checkInterrupted(); + blocks[i] = readRowStripePixels(src, resolution, batchStart + i, allRowsStartIndices); + } + final var sortedBatch = sortStripeBatch(blocks, stripeExecutor); + + for (int i = 0; i < localCount; i++) { + checkInterrupted(); + final int rowStripe = batchStart + i; + final var sorted = sortedBatch[i]; + if (sorted != null) { + final var colStripes = sorted.colStripes(); + int start = 0; + while (start < colStripes.length) { + int end = start + 1; + while (end < colStripes.length && colStripes[end] == colStripes[start]) { + end++; + } + final int blockLen = end - start; + if (blockLen >= DENSE_THRESHOLD) { + denseCount++; + } else { + sparseCount += blockLen; + } + start = end; + } + } + countingProgressReporter.accept(rowStripe + 1); + } + } + } finally { + if (stripeExecutor != null) { + stripeExecutor.shutdownNow(); + } + } + + return new StripeCounts(sparseCount, denseCount); + } + + private static @NotNull SortedStripePixels[] sortStripeBatch( + final PixelBlock @NotNull [] blocks, + final ExecutorService stripeExecutor + ) { + final var sortedBatch = new SortedStripePixels[blocks.length]; + if (stripeExecutor == null) { + for (int i = 0; i < blocks.length; i++) { + checkInterrupted(); + final var block = blocks[i]; + if (block.length() > 0) { + sortedBatch[i] = sortStripePixels(block.rows(), block.cols(), block.values()); + } + } + return sortedBatch; + } + + final List> futures = new ArrayList<>(blocks.length); + for (int i = 0; i < blocks.length; i++) { + final int idx = i; + futures.add(stripeExecutor.submit(() -> { + checkInterrupted(); + final var block = blocks[idx]; + if (block.length() > 0) { + sortedBatch[idx] = sortStripePixels(block.rows(), block.cols(), block.values()); + } + })); + } + + try { + for (final var f : futures) { + f.get(); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } + return sortedBatch; + } + + private static long estimateEtaMillis(final long done, final long total, final long elapsedMillis) { + if (done <= 0 || total <= 0 || done >= total || elapsedMillis <= 0) { + return 0L; + } + final var remaining = total - done; + return (elapsedMillis * remaining) / done; + } + + private static int resolveRequestedWorkers(final int parallelismOption) { + if (parallelismOption == -1 || parallelismOption <= 0) { + return Math.max(1, Runtime.getRuntime().availableProcessors()); + } + return parallelismOption; + } + + private static @NotNull String formatDuration(final long millis) { + if (millis <= 0) { + return "00:00"; + } + final long totalSeconds = millis / 1000L; + final long hours = totalSeconds / 3600L; + final long minutes = (totalSeconds % 3600L) / 60L; + final long seconds = totalSeconds % 60L; + if (hours > 0) { + return String.format("%02d:%02d:%02d", hours, minutes, seconds); + } + return String.format("%02d:%02d", minutes, seconds); + } + + private static void checkInterrupted() { + if (Thread.currentThread().isInterrupted()) { + throw new RuntimeException("Conversion interrupted"); + } + } + + private static @NotNull PixelBlock readRowStripePixels( + final @NotNull IHDF5Reader src, + final long resolution, + final int rowStripe, + final long @NotNull [] allRowsStartIndices + ) { + final long startOffset = allRowsStartIndices[rowStripe * SUBMATRIX_SIZE]; + final int nextBin = Math.min((rowStripe + 1) * SUBMATRIX_SIZE, allRowsStartIndices.length - 1); + final long endOffset = allRowsStartIndices[nextBin]; + final int length = (int) (endOffset - startOffset); + + if (length <= 0) { + return new PixelBlock(new long[0], new long[0], new long[0]); + } + + final var base = "/resolutions/" + resolution + "/pixels/"; + final var rows = src.int64().readArrayBlockWithOffset(base + "bin1_id", length, startOffset); + final var cols = src.int64().readArrayBlockWithOffset(base + "bin2_id", length, startOffset); + final var vals = src.int64().readArrayBlockWithOffset(base + "count", length, startOffset); + return new PixelBlock(rows, cols, vals); + } + + private static @NotNull SortedStripePixels sortStripePixels( + final long @NotNull [] rows, + final long @NotNull [] cols, + final long @NotNull [] values + ) { + final int n = rows.length; + final var order = new Integer[n]; + for (int i = 0; i < n; i++) { + order[i] = i; + } + + Arrays.sort(order, Comparator + .comparingLong((Integer i) -> cols[i] / SUBMATRIX_SIZE) + .thenComparingLong(i -> rows[i] / SUBMATRIX_SIZE) + .thenComparingLong(i -> rows[i] % SUBMATRIX_SIZE) + .thenComparingLong(i -> cols[i] % SUBMATRIX_SIZE) + ); + + final var sortedColStripes = new long[n]; + final var sortedIntraRows = new int[n]; + final var sortedIntraCols = new int[n]; + final var sortedValues = new long[n]; + + for (int i = 0; i < n; i++) { + final int srcIdx = order[i]; + sortedColStripes[i] = cols[srcIdx] / SUBMATRIX_SIZE; + sortedIntraRows[i] = (int) (rows[srcIdx] % SUBMATRIX_SIZE); + sortedIntraCols[i] = (int) (cols[srcIdx] % SUBMATRIX_SIZE); + sortedValues[i] = values[srcIdx]; + } + + return new SortedStripePixels(sortedColStripes, sortedIntraRows, sortedIntraCols, sortedValues); + } + + private static @NotNull List dumpStripeData( + final @NotNull IHDF5Reader src, + final @NotNull IHDF5Writer dst, + final long resolution, + final @NotNull String nameLengthPath, + final @NotNull HDF5FloatStorageFeatures floatStorageFeatures + ) { + final var chromLengthPath = nameLengthPath + "/length"; + final var chromNamePath = nameLengthPath + "/name"; + + final long[] chromLengths = src.int64().readArray(chromLengthPath); + final String[] chromNames = src.string().readArray(chromNamePath); + if (chromLengths.length != chromNames.length) { + throw new IllegalStateException("Chromosome lengths and names have different sizes at " + nameLengthPath); + } + + final var stripesRoot = "/resolutions/" + resolution + "/stripes"; + dst.object().createGroup(stripesRoot); + + final var binsRoot = "/resolutions/" + resolution + "/bins"; + final long binCount = datasetLength(src, binsRoot + "/chrom"); + final boolean hasWeights = src.object().isDataSet(binsRoot + "/weight"); + + final int stripeCount = (int) ((binCount / SUBMATRIX_SIZE) + Math.min(binCount % SUBMATRIX_SIZE, 1L)); + final List stripes = new ArrayList<>(stripeCount); + final var stripeLengths = new long[stripeCount]; + final var stripeWeights = new double[stripeCount][SUBMATRIX_SIZE]; + + for (int stripeId = 0; stripeId < stripeCount; stripeId++) { + final long start = (long) stripeId * SUBMATRIX_SIZE; + final long end = Math.min((long) (stripeId + 1) * SUBMATRIX_SIZE, binCount); + final int stripeLen = (int) (end - start); + + final double[] weights; + if (hasWeights) { + weights = src.float64().readArrayBlockWithOffset(binsRoot + "/weight", stripeLen, start); + } else { + weights = new double[SUBMATRIX_SIZE]; + Arrays.fill(weights, 1.0d); + } + + final var padded = new double[SUBMATRIX_SIZE]; + Arrays.fill(padded, 1.0d); + System.arraycopy(weights, 0, padded, 0, Math.min(weights.length, SUBMATRIX_SIZE)); + + stripeLengths[stripeId] = stripeLen; + stripeWeights[stripeId] = padded; + stripes.add(new StripeDescriptor(stripeId, stripeLen, weights)); + } + + dst.int64().writeArray(getStripeLengthsBinsDatasetPath(resolution), stripeLengths, HDF5IntStorageFeatures.INT_CHUNKED); + dst.float64().writeMatrix(getStripeBinWeightsDatasetPath(resolution), stripeWeights, floatStorageFeatures); + + return stripes; + } + + private static void dumpContigData( + final @NotNull IHDF5Reader src, + final @NotNull IHDF5Writer dst, + final @NotNull List resolutions, + final int parallelism, + final @NotNull HDF5IntStorageFeatures intStorageFeatures, + final @NotNull HDF5FloatStorageFeatures floatStorageFeatures + ) { + final long anyResolution = resolutions.get(0); + final String nameLengthPath = resolveNameLengthPath(src, anyResolution); + + dst.object().createGroup("/contig_info"); + final String[] contigNames = src.string().readArray(nameLengthPath + "/name"); + final int contigCount = contigNames.length; + dst.string().writeArray(getContigNameDatasetPath(), contigNames); + + final long[] contigDirections = new long[contigCount]; + Arrays.fill(contigDirections, ContigDirection.FORWARD.ordinal()); + dst.int64().writeArray(getContigDirectionDatasetPath(), contigDirections, intStorageFeatures); + + final long[] orderedContigIds = new long[contigCount]; + final long[] contigScaffoldIds = new long[contigCount]; + Arrays.fill(contigScaffoldIds, -1L); + for (int i = 0; i < contigCount; i++) { + orderedContigIds[i] = i; + } + dst.int64().writeArray(getContigOrderDatasetPath(), orderedContigIds, intStorageFeatures); + dst.int64().writeArray("/contig_info/contig_scaffold_id", contigScaffoldIds, intStorageFeatures); + + final long[] contigLengthBp; + if (src.object().isDataSet(nameLengthPath + "/length")) { + contigLengthBp = src.int64().readArray(nameLengthPath + "/length"); + } else { + final var chromOffsets = src.int64().readArray("/resolutions/" + anyResolution + "/indexes/chrom_offset"); + final var binEnds = src.int64().readArray("/resolutions/" + anyResolution + "/bins/end"); + contigLengthBp = new long[contigCount]; + for (int i = 0; i < contigCount - 1; i++) { + contigLengthBp[i] = binEnds[(int) (chromOffsets[i + 1] - 1L)]; + } + contigLengthBp[contigCount - 1] = binEnds[binEnds.length - 1]; + } + dst.int64().writeArray(getContigLengthBpDatasetPath(), contigLengthBp, intStorageFeatures); + + final Map contigStartBinsByResolution = new HashMap<>(); + final Map contigLengthBinsByResolution = new HashMap<>(); + final Map> stripesByResolution = new HashMap<>(); + + for (final var resolution : resolutions) { + final var chromOffsets = src.int64().readArray("/resolutions/" + resolution + "/indexes/chrom_offset"); + final var lengthBins = new long[chromOffsets.length - 1]; + for (int i = 0; i < lengthBins.length; i++) { + lengthBins[i] = chromOffsets[i + 1] - chromOffsets[i]; + if (lengthBins[i] <= 0) { + throw new IllegalStateException("Zero-length contig found at resolution " + resolution + " contig=" + i); + } + } + + contigStartBinsByResolution.put(resolution, chromOffsets); + contigLengthBinsByResolution.put(resolution, lengthBins); + final var path = resolveNameLengthPath(src, resolution); + stripesByResolution.put(resolution, buildStripeDescriptorsOnly(src, resolution, path)); + } + + for (final var resolution : resolutions) { + final var resolutionRoot = "/resolutions/" + resolution; + dst.object().createGroup(resolutionRoot + "/contigs"); + dst.object().createGroup(resolutionRoot + "/atl"); + + final var contigLengthBins = contigLengthBinsByResolution.get(resolution); + final var hideTypes = new byte[contigCount]; + for (int i = 0; i < contigCount; i++) { + hideTypes[i] = (byte) ((contigLengthBins[i] > 1L) ? ContigHideType.SHOWN.ordinal() : ContigHideType.HIDDEN.ordinal()); + } + + dst.int64().writeArray(getContigLengthBinsDatasetPath(resolution), contigLengthBins, intStorageFeatures); + dst.int8().writeArray(getContigHideTypeDatasetPath(resolution), hideTypes); + + final AtomicReferenceArray> atusByContig = new AtomicReferenceArray<>(contigCount); + runParallelFor(parallelism, contigCount, contigId -> { + final var atus = generateAtusForContig( + contigId, + resolution, + contigStartBinsByResolution, + contigLengthBinsByResolution, + stripesByResolution + ); + atusByContig.set(contigId, atus); + }); + + long totalAtuCount = 0L; + for (int i = 0; i < contigCount; i++) { + totalAtuCount += Objects.requireNonNull(atusByContig.get(i)).size(); + } + + final long[][] basisAtu = new long[(int) totalAtuCount][4]; + final long[][] contigsAtl = new long[(int) totalAtuCount][2]; + + int atuCursor = 0; + for (int contigId = 0; contigId < contigCount; contigId++) { + final var atus = atusByContig.get(contigId); + for (int i = 0; i < atus.size(); i++) { + final var atu = atus.get(i); + contigsAtl[atuCursor][0] = contigId; + contigsAtl[atuCursor][1] = atuCursor; + + basisAtu[atuCursor][0] = atu.getStripeDescriptor().stripeId(); + basisAtu[atuCursor][1] = atu.getStartIndexInStripeIncl(); + basisAtu[atuCursor][2] = atu.getEndIndexInStripeExcl(); + basisAtu[atuCursor][3] = atu.getDirection().ordinal(); + + atuCursor++; + } + } + + dst.int64().writeMatrix(getContigsATLDatasetPath(resolution), contigsAtl); + dst.int64().writeMatrix(getBasisATUDatasetPath(resolution), basisAtu); + } + } + + private static @NotNull List generateAtusForContig( + final int contigId, + final long resolution, + final @NotNull Map contigStartBinsByResolution, + final @NotNull Map contigLengthBinsByResolution, + final @NotNull Map> stripesByResolution + ) { + long startBin = contigStartBinsByResolution.get(resolution)[contigId]; + final long endBin = startBin + contigLengthBinsByResolution.get(resolution)[contigId]; + final long startStripeId = startBin / SUBMATRIX_SIZE; + + final var stripes = stripesByResolution.get(resolution); + final var atus = new ArrayList(); + + atus.add(new ATUDescriptor( + stripes.get((int) startStripeId), + (int) (startBin % SUBMATRIX_SIZE), + (int) (((startBin / SUBMATRIX_SIZE) < (endBin / SUBMATRIX_SIZE)) ? SUBMATRIX_SIZE : (1 + ((endBin - 1L) % SUBMATRIX_SIZE))), + ATUDirection.FORWARD + )); + + startBin = ((startBin / SUBMATRIX_SIZE) + 1L) * SUBMATRIX_SIZE; + final long equalPartsCount = (endBin - startBin) / 256L; + + for (int part = 0; part < equalPartsCount; part++) { + atus.add(new ATUDescriptor( + stripes.get((int) (startStripeId + part + 1L)), + 0, + SUBMATRIX_SIZE, + ATUDirection.FORWARD + )); + } + + startBin += (long) (atus.size() - 1) * SUBMATRIX_SIZE; + + if (startBin < endBin) { + atus.add(new ATUDescriptor( + stripes.get((int) (startStripeId + 1L + equalPartsCount)), + 0, + (int) (1L + ((endBin - 1L) % SUBMATRIX_SIZE)), + ATUDirection.FORWARD + )); + } + + return atus; + } + + private static @NotNull List buildStripeDescriptorsOnly( + final @NotNull IHDF5Reader src, + final long resolution, + final @NotNull String nameLengthPath + ) { + final var chromLengthPath = nameLengthPath + "/length"; + final var chromNamePath = nameLengthPath + "/name"; + + final long[] chromLengths = src.int64().readArray(chromLengthPath); + final String[] chromNames = src.string().readArray(chromNamePath); + if (chromLengths.length != chromNames.length) { + throw new IllegalStateException("Chromosome lengths and names have different sizes at " + nameLengthPath); + } + + final var binsRoot = "/resolutions/" + resolution + "/bins"; + final long binCount = datasetLength(src, binsRoot + "/chrom"); + final boolean hasWeights = src.object().isDataSet(binsRoot + "/weight"); + + final int stripeCount = (int) ((binCount / SUBMATRIX_SIZE) + Math.min(binCount % SUBMATRIX_SIZE, 1L)); + final List stripes = new ArrayList<>(stripeCount); + + for (int stripeId = 0; stripeId < stripeCount; stripeId++) { + final long start = (long) stripeId * SUBMATRIX_SIZE; + final long end = Math.min((long) (stripeId + 1) * SUBMATRIX_SIZE, binCount); + final int stripeLen = (int) (end - start); + + final double[] weights; + if (hasWeights) { + weights = src.float64().readArrayBlockWithOffset(binsRoot + "/weight", stripeLen, start); + } else { + weights = new double[SUBMATRIX_SIZE]; + Arrays.fill(weights, 1.0d); + } + + stripes.add(new StripeDescriptor(stripeId, stripeLen, weights)); + } + + return stripes; + } + + private static void runParallelFor(final int parallelism, final int itemCount, final @NotNull java.util.function.IntConsumer task) { + if (parallelism <= 1 || itemCount <= 1) { + for (int i = 0; i < itemCount; i++) { + task.accept(i); + } + return; + } + + final var pool = Executors.newFixedThreadPool(parallelism); + try { + final List> futures = new ArrayList<>(itemCount); + for (int i = 0; i < itemCount; i++) { + final int idx = i; + futures.add(pool.submit(() -> task.accept(idx))); + } + for (final var f : futures) { + f.get(); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + throw new RuntimeException(e.getCause()); + } finally { + pool.shutdownNow(); + } + } + + private static @NotNull String resolveNameLengthPath(final @NotNull IHDF5Reader src, final long resolution) { + final var perResolution = "/resolutions/" + resolution + "/chroms"; + if (src.object().isGroup(perResolution) && src.object().isDataSet(perResolution + "/name") && src.object().isDataSet(perResolution + "/length")) { + return perResolution; + } + + final var global = "/chroms"; + if (src.object().isGroup(global) && src.object().isDataSet(global + "/name") && src.object().isDataSet(global + "/length")) { + return global; + } + + throw new IllegalStateException("Cannot resolve chromosome name/length path for resolution " + resolution); + } + + private static long datasetLength(final @NotNull IHDF5Reader src, final @NotNull String path) { + final var dims = src.object().getDataSetInformation(path).getDimensions(); + return dims.length == 0 ? 0L : dims[0]; + } + + private static int safeChunkLen(final long length, final int preferred) { + final long base = Math.max(1L, Math.min(Math.max(1L, (long) preferred), Math.max(1L, length))); + return (int) Math.min(base, Integer.MAX_VALUE); + } + + private @NotNull List resolveResolutions(final @NotNull IHDF5Reader src, final @NotNull List requested) { + final var available = src.object().getAllGroupMembers("/resolutions").stream().map(s -> { + try { + return Long.parseLong(s); + } catch (NumberFormatException ignored) { + return null; + } + }).filter(Objects::nonNull).toList(); + + if (requested == null || requested.isEmpty()) { + return available; + } + + final var availableSet = new java.util.HashSet<>(available); + return requested.stream().filter(availableSet::contains).toList(); + } + + @FunctionalInterface + private interface StripeProgressReporter { + void report(int processedStripes, int stripeCount); + } + + private static final class ConversionProgressTracker { + private final int totalSteps; + private final @NotNull Consumer logger; + private final @NotNull AtomicInteger completedSteps = new AtomicInteger(0); + private final long startedNanos = System.nanoTime(); + + private ConversionProgressTracker(final int totalSteps, final @NotNull Consumer logger) { + this.totalSteps = Math.max(1, totalSteps); + this.logger = logger; + } + + private void markStep(final @NotNull String stepDescription) { + final int done = completedSteps.incrementAndGet(); + final int percent = (int) ((done * 100L) / totalSteps); + final long elapsedMillis = (System.nanoTime() - startedNanos) / 1_000_000L; + final long etaMillis = estimateEtaMillis(done, totalSteps, elapsedMillis); + logger.accept( + String.format( + "Overall progress: %d%% (%d/%d), elapsed=%s, eta=%s - %s", + percent, + done, + totalSteps, + formatDuration(elapsedMillis), + formatDuration(etaMillis), + stepDescription + ) + ); + } + } + + private static final class PhaseProgressTracker { + private final @NotNull String label; + private final int totalItems; + private final @NotNull Consumer logger; + private final long startedNanos = System.nanoTime(); + private int lastLoggedPercent = -1; + + private PhaseProgressTracker( + final @NotNull String label, + final int totalItems, + final @NotNull Consumer logger + ) { + this.label = label; + this.totalItems = Math.max(0, totalItems); + this.logger = logger; + if (this.totalItems == 0) { + logger.accept(label + ": 100% (0/0), elapsed=00:00, eta=00:00"); + } + } + + private void report(final int doneItems) { + if (totalItems <= 0) { + return; + } + final int clampedDone = Math.max(0, Math.min(doneItems, totalItems)); + final int percent = (int) ((clampedDone * 100L) / totalItems); + if (lastLoggedPercent >= 0 && percent < 100 && percent - lastLoggedPercent < 5) { + return; + } + lastLoggedPercent = percent; + final long elapsedMillis = (System.nanoTime() - startedNanos) / 1_000_000L; + final long etaMillis = estimateEtaMillis(clampedDone, totalItems, elapsedMillis); + logger.accept( + String.format( + "%s: %d%% (%d/%d), elapsed=%s, eta=%s", + label, + percent, + clampedDone, + totalItems, + formatDuration(elapsedMillis), + formatDuration(etaMillis) + ) + ); + } + + private void finish() { + report(totalItems); + } + } + + private static final SortedStripePixels EMPTY_STRIPE = + new SortedStripePixels(new long[0], new int[0], new int[0], new long[0]); + + private record PixelBlock(long @NotNull [] rows, long @NotNull [] cols, long @NotNull [] values) { + int length() { + return rows.length; + } + } + + private record SortedStripePixels( + long @NotNull [] colStripes, + int @NotNull [] intraRows, + int @NotNull [] intraCols, + long @NotNull [] values + ) { + } + + private record StripeCounts(long sparseElementCount, long denseBlockCount) { + } + + private record SaveBlockResult(long sparseOffset, long denseOffset) { + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDescriptor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDescriptor.java index 17f447b..0c28885 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDescriptor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDescriptor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDirection.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDirection.java index 2810065..5eb7f70 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDirection.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ATUDirection.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/AssemblyInfo.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/AssemblyInfo.java index 94297bc..44451da 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/AssemblyInfo.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/AssemblyInfo.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDescriptor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDescriptor.java index 1676606..400a7cf 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDescriptor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDescriptor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDirection.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDirection.java index 3d8a458..340fdbd 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDirection.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigDirection.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigHideType.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigHideType.java index 57e6041..4419535 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigHideType.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ContigHideType.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/QueryLengthUnit.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/QueryLengthUnit.java index 0fcddbb..c28e91d 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/QueryLengthUnit.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/QueryLengthUnit.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ScaffoldDescriptor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ScaffoldDescriptor.java index b7d91e0..7db3b5d 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ScaffoldDescriptor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/ScaffoldDescriptor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/StripeDescriptor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/StripeDescriptor.java index 5587fce..babbfff 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/StripeDescriptor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/domain/StripeDescriptor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ContigTree.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ContigTree.java index 369d011..99a6853 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ContigTree.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ContigTree.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ScaffoldTree.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ScaffoldTree.java index 0d7c945..acdf941 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ScaffoldTree.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/trees/ScaffoldTree.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/util/BinarySearch.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/util/BinarySearch.java index edb4469..8929c58 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/util/BinarySearch.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/util/BinarySearch.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/util/CommonUtils.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/util/CommonUtils.java index e06d14e..e37d930 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/util/CommonUtils.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/util/CommonUtils.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/util/matrix/SparseCOOMatrixLong.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/util/matrix/SparseCOOMatrixLong.java index 06b1ec7..3ec968e 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/util/matrix/SparseCOOMatrixLong.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/util/matrix/SparseCOOMatrixLong.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/RawTileWithWeights.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/RawTileWithWeights.java index 9602858..6e026c6 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/RawTileWithWeights.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/RawTileWithWeights.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/SimpleVisualizationOptions.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/SimpleVisualizationOptions.java index 814ca12..7ac2163 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/SimpleVisualizationOptions.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/SimpleVisualizationOptions.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileVisualizationProcessor.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileVisualizationProcessor.java index c158a7b..2f3f81d 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileVisualizationProcessor.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileVisualizationProcessor.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileWithWeights.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileWithWeights.java index d26c50e..d44f9a3 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileWithWeights.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/TileWithWeights.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/Colormap.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/Colormap.java index 54fec37..413b572 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/Colormap.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/Colormap.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/DoubleColormap.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/DoubleColormap.java index 27af3af..38d7cae 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/DoubleColormap.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/DoubleColormap.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/gradient/SimpleLinearGradient.java b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/gradient/SimpleLinearGradient.java index 17bf669..6bf3d48 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/gradient/SimpleLinearGradient.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_library/visualization/colormap/gradient/SimpleLinearGradient.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/HandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/HandlersHolder.java index 9a274f4..52e1b7b 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/HandlersHolder.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/HandlersHolder.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/MainVerticle.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/MainVerticle.java index 1b40068..f4125ec 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/MainVerticle.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/MainVerticle.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -30,23 +30,30 @@ import io.vertx.config.ConfigRetrieverOptions; import io.vertx.config.ConfigStoreOptions; import io.vertx.core.AbstractVerticle; +import io.vertx.core.Launcher; import io.vertx.core.Promise; import io.vertx.core.Vertx; import io.vertx.core.http.HttpServerOptions; import io.vertx.core.json.JsonArray; import io.vertx.core.json.JsonObject; import io.vertx.core.logging.SLF4JLogDelegateFactory; +import io.vertx.core.shareddata.LocalMap; import io.vertx.ext.web.Router; import io.vertx.ext.web.handler.BodyHandler; import io.vertx.ext.web.handler.CorsHandler; +import lombok.NonNull; import lombok.extern.slf4j.Slf4j; +import org.jetbrains.annotations.NotNull; import org.slf4j.LoggerFactory; import ru.itmo.ctlab.hict.hict_library.chunkedfile.hdf5.HDF5LibraryInitializer; import ru.itmo.ctlab.hict.hict_library.visualization.SimpleVisualizationOptions; import ru.itmo.ctlab.hict.hict_library.visualization.colormap.gradient.SimpleLinearGradient; import ru.itmo.ctlab.hict.hict_server.handlers.fileop.FileOpHandlersHolder; import ru.itmo.ctlab.hict.hict_server.handlers.files.FSHandlersHolder; +import ru.itmo.ctlab.hict.hict_server.handlers.names.NameMappingHandlersHolder; import ru.itmo.ctlab.hict.hict_server.handlers.operations.ScaffoldingOpHandlersHolder; +import ru.itmo.ctlab.hict.hict_server.handlers.conversion.ConversionHandlersHolder; +import ru.itmo.ctlab.hict.hict_server.handlers.info.InfoHandlersHolder; import ru.itmo.ctlab.hict.hict_server.handlers.tiles.TileHandlersHolder; import ru.itmo.ctlab.hict.hict_server.util.shareable.ShareableWrappers; @@ -64,6 +71,9 @@ public class MainVerticle extends AbstractVerticle { HDF5LibraryInitializer.initializeHDF5Library(); } + public static void main(final String[] args) { + Launcher.executeCommand("run", MainVerticle.class.getName()); + } @Override public void start(final Promise startPromise) throws Exception { @@ -76,11 +86,11 @@ public void start(final Promise startPromise) throws Exception { final Logger root = (Logger) LoggerFactory.getLogger(org.slf4j.Logger.ROOT_LOGGER_NAME); root.setLevel(Level.INFO); - log.info("Logging initialized"); final ConfigStoreOptions jsonEnvConfig = new ConfigStoreOptions().setType("env") - .setConfig(new JsonObject().put("keys", new JsonArray().add("DATA_DIR").add("TILE_SIZE").add("VXPORT").add("MIN_DS_POOL").add("MAX_DS_POOL"))); + .setConfig(new JsonObject().put("keys", + new JsonArray().add("DATA_DIR").add("TILE_SIZE").add("VXPORT").add("MIN_DS_POOL").add("MAX_DS_POOL"))); final ConfigRetrieverOptions myOptions = new ConfigRetrieverOptions().addStore(jsonEnvConfig); final ConfigRetriever myConfigRetriver = ConfigRetriever.create(vertx, myOptions); myConfigRetriver.getConfig(asyncResults -> System.out.println(asyncResults.result().encodePrettily())); @@ -96,7 +106,7 @@ public void start(final Promise startPromise) throws Exception { try { log.info("Trying to write configuration to local map"); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); map.put("dataDirectory", new ShareableWrappers.PathWrapper(dataDirectory)); map.put("tileSize", tileSize); map.put("VXPORT", port); @@ -104,15 +114,15 @@ public void start(final Promise startPromise) throws Exception { map.put("MAX_DS_POOL", maxDSPool); final var defaultVisualizationOptions = new SimpleVisualizationOptions(10.0, 0.0, false, false, false, - new SimpleLinearGradient( - 32, - new Color(255, 255, 255, 0), - new Color(0, 96, 0, 255), - 0.0d, - 1.0d - )); + new SimpleLinearGradient( + 32, + new Color(255, 255, 255, 0), + new Color(0, 96, 0, 255), + 0.0d, + 1.0d)); - map.put("visualizationOptions", new ShareableWrappers.SimpleVisualizationOptionsWrapper(defaultVisualizationOptions)); + map.put("visualizationOptions", + new ShareableWrappers.SimpleVisualizationOptionsWrapper(defaultVisualizationOptions)); log.info("Added to local map"); } finally { @@ -135,16 +145,16 @@ public void start(final Promise startPromise) throws Exception { final var router = Router.router(vertx); router.route().handler(CorsHandler.create() - .allowedMethod(io.vertx.core.http.HttpMethod.GET) - .allowedMethod(io.vertx.core.http.HttpMethod.POST) - .allowedMethod(io.vertx.core.http.HttpMethod.OPTIONS) - .allowedHeader("Access-Control-Request-Method") - .allowedHeader("Access-Control-Allow-Credentials") - .allowedHeader("Access-Control-Allow-Origin") - .allowedHeader("Access-Control-Allow-Headers") - .allowedHeader("Content-Type")); - router.route().handler(BodyHandler.create()); -// router.route().handler(ErrorHandler.create(Vertx.vertx())); + .allowedMethod(io.vertx.core.http.HttpMethod.GET) + .allowedMethod(io.vertx.core.http.HttpMethod.POST) + .allowedMethod(io.vertx.core.http.HttpMethod.OPTIONS) + .allowedHeader("Access-Control-Request-Method") + .allowedHeader("Access-Control-Allow-Credentials") + .allowedHeader("Access-Control-Allow-Origin") + .allowedHeader("Access-Control-Allow-Headers") + .allowedHeader("Content-Type")); + router.route().handler(BodyHandler.create().setUploadsDirectory("/tmp").setBodyLimit(2L * 1024 * 1024 * 1024)); + // router.route().handler(ErrorHandler.create(Vertx.vertx())); vertx.exceptionHandler(event -> { log.error("An exception was caught at the top level", event); log.debug(event.getMessage()); @@ -154,7 +164,7 @@ public void start(final Promise startPromise) throws Exception { final int port; try { - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); port = (int) map.get("VXPORT"); } finally { log.info("Finished maps"); @@ -165,23 +175,22 @@ public void start(final Promise startPromise) throws Exception { Vertx.currentContext().exceptionHandler().handle(err); }); - log.info("Initializing handlers"); final List handlersHolders = new ArrayList<>(); handlersHolders.add(new FSHandlersHolder(vertx)); handlersHolders.add(new TileHandlersHolder(vertx)); handlersHolders.add(new FileOpHandlersHolder(vertx)); handlersHolders.add(new ScaffoldingOpHandlersHolder(vertx)); - + handlersHolders.add(new NameMappingHandlersHolder(vertx)); + handlersHolders.add(new ConversionHandlersHolder(vertx)); + handlersHolders.add(new InfoHandlersHolder(vertx)); router.route().failureHandler(ctx -> { log.error("An exception was caught at router top-level", ctx.failure()); ctx.response().end( - ctx.failure().getMessage() - ); + ctx.failure().getMessage()); }); - log.info("Configuring router"); handlersHolders.forEach(handlersHolder -> handlersHolder.addHandlersToRouter(router)); @@ -190,6 +199,12 @@ public void start(final Promise startPromise) throws Exception { log.info("Server started"); log.info("Deploying WebUI Verticle"); - vertx.deployVerticle(new WebUIVerticle()); + vertx.deployVerticle(new WebUIVerticle(), ar -> { + if (ar.succeeded()) { + log.info("WebUI verticle deployed"); + } else { + log.error("WebUI verticle deployment failed", ar.cause()); + } + }); } } diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/WebUIVerticle.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/WebUIVerticle.java index 71000c7..11fe147 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/WebUIVerticle.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/WebUIVerticle.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -33,11 +33,17 @@ import io.vertx.core.json.JsonArray; import io.vertx.core.json.JsonObject; import io.vertx.core.logging.SLF4JLogDelegateFactory; +import io.vertx.core.shareddata.LocalMap; import io.vertx.ext.web.Router; import io.vertx.ext.web.handler.CorsHandler; +import io.vertx.ext.web.handler.FileSystemAccess; import io.vertx.ext.web.handler.StaticHandler; +import lombok.NonNull; import lombok.extern.slf4j.Slf4j; +import org.jetbrains.annotations.NotNull; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.concurrent.BrokenBarrierException; import java.util.concurrent.CyclicBarrier; @@ -47,6 +53,7 @@ public class WebUIVerticle extends AbstractVerticle { @Override public void start(final Promise startPromise) throws Exception { + try { // set vertx logger delegate factory to slf4j String logFactory = System.getProperty("org.vertx.logger-delegate-factory-class-name"); if (logFactory == null) { @@ -63,12 +70,12 @@ public void start(final Promise startPromise) throws Exception { final CyclicBarrier barrier = new CyclicBarrier(1); myConfigRetriver.getConfig(event -> { - final var serveWebUI = event.result().getBoolean("SERVE_WEBUI", true); - final var webuiPort = event.result().getInteger("WEBUI_PORT", 8080); + final var serveWebUI = resolveServeWebUI(event.result()); + final var webuiPort = resolveWebuiPort(event.result()); try { log.info("Trying to write WebUI configuration to local map"); - final var map = vertx.sharedData().getLocalMap("webui_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("webui_server"); map.put("WEBUI_PORT", webuiPort); map.put("SERVE_WEBUI", serveWebUI); log.info("Added to local map"); @@ -109,7 +116,7 @@ public void start(final Promise startPromise) throws Exception { final int webuiPort; final boolean serve; try { - final var map = vertx.sharedData().getLocalMap("webui_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("webui_server"); serve = (boolean) map.get("SERVE_WEBUI"); webuiPort = (int) map.get("WEBUI_PORT"); } finally { @@ -125,12 +132,71 @@ public void start(final Promise startPromise) throws Exception { log.info("WebUI Server will start on port " + webuiPort); - webuiRouter.route("/").handler(ctx -> ctx.response().sendFile("webui/index.html")); - webuiRouter.route("/*").handler(StaticHandler.create("webui")); - + final var webuiStaticHandler = createWebuiStaticHandler(); + webuiRouter.route("/").handler(ctx -> ctx.reroute("/index.html")); + webuiRouter.route("/*").handler(webuiStaticHandler); + log.info("WebUI router configured, binding HTTP server"); log.info("Starting WebUI server on port " + webuiPort); - webuiServer.requestHandler(webuiRouter).listen(webuiPort); - log.info("WebUI Server started"); + webuiServer.requestHandler(webuiRouter).listen(webuiPort, "0.0.0.0", ar -> { + if (ar.succeeded()) { + log.info("WebUI Server started on 0.0.0.0:{}", webuiServer.actualPort()); + startPromise.complete(); + } else { + log.error("Failed to start WebUI server on port " + webuiPort, ar.cause()); + startPromise.fail(ar.cause()); + } + }); + } catch (Throwable t) { + log.error("WebUI verticle start failed", t); + startPromise.fail(t); + } + } + + private boolean resolveServeWebUI(final @NotNull JsonObject config) { + final var systemOverride = System.getProperty("SERVE_WEBUI"); + if (systemOverride != null && !systemOverride.isBlank()) { + return Boolean.parseBoolean(systemOverride.trim()); + } + return config.getBoolean("SERVE_WEBUI", true); + } + + private int resolveWebuiPort(final @NotNull JsonObject config) { + final var systemOverride = System.getProperty("WEBUI_PORT"); + if (systemOverride != null && !systemOverride.isBlank()) { + try { + return Integer.parseInt(systemOverride.trim()); + } catch (NumberFormatException ignored) { + log.warn("Invalid WEBUI_PORT system property: {}", systemOverride); + } + } + return config.getInteger("WEBUI_PORT", 8080); + } + + private @NotNull StaticHandler createWebuiStaticHandler() { + final var explicitRoot = System.getenv("WEBUI_ROOT"); + if (explicitRoot != null && !explicitRoot.isBlank()) { + final var explicitRootPath = Path.of(explicitRoot).toAbsolutePath().normalize(); + if (Files.isDirectory(explicitRootPath)) { + log.info("Serving WebUI from WEBUI_ROOT={}", explicitRootPath); + return StaticHandler.create(FileSystemAccess.ROOT, explicitRootPath.toString()); + } + log.warn("WEBUI_ROOT is set but does not exist: {}", explicitRootPath); + } + + final var localDist = Path.of("../HiCT_WebUI/dist").toAbsolutePath().normalize(); + if (Files.isDirectory(localDist)) { + log.info("Serving WebUI from local checkout: {}", localDist); + return StaticHandler.create(FileSystemAccess.ROOT, localDist.toString()); + } + + final var builtCloneDist = Path.of("build/webui/HiCT_WebUI/dist").toAbsolutePath().normalize(); + if (Files.isDirectory(builtCloneDist)) { + log.info("Serving WebUI from gradle clone output: {}", builtCloneDist); + return StaticHandler.create(FileSystemAccess.ROOT, builtCloneDist.toString()); + } + + log.info("Serving WebUI from classpath resources: webui"); + return StaticHandler.create("webui"); } } diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/ImportNameMappingRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/ImportNameMappingRequestDTO.java new file mode 100644 index 0000000..2c85056 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/ImportNameMappingRequestDTO.java @@ -0,0 +1,66 @@ +/* + * MIT License + * + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package ru.itmo.ctlab.hict.hict_server.dto.request.names; + +import io.vertx.core.json.JsonArray; +import io.vertx.core.json.JsonObject; +import org.jetbrains.annotations.NotNull; + +import java.util.ArrayList; +import java.util.List; + +public record ImportNameMappingRequestDTO( + @NotNull List<@NotNull ContigNameMappingRequestDTO> contigs, + @NotNull List<@NotNull ScaffoldNameMappingRequestDTO> scaffolds +) { + public static @NotNull ImportNameMappingRequestDTO fromJSONObject(final @NotNull JsonObject jsonObject) { + return new ImportNameMappingRequestDTO( + parseContigs(jsonObject.getJsonArray("contigs", new JsonArray())), + parseScaffolds(jsonObject.getJsonArray("scaffolds", new JsonArray())) + ); + } + + private static @NotNull List<@NotNull ContigNameMappingRequestDTO> parseContigs(final @NotNull JsonArray array) { + final var result = new ArrayList(array.size()); + for (int i = 0; i < array.size(); i++) { + final var obj = array.getJsonObject(i); + result.add(new ContigNameMappingRequestDTO(obj.getInteger("contigId"), obj.getString("name"))); + } + return result; + } + + private static @NotNull List<@NotNull ScaffoldNameMappingRequestDTO> parseScaffolds(final @NotNull JsonArray array) { + final var result = new ArrayList(array.size()); + for (int i = 0; i < array.size(); i++) { + final var obj = array.getJsonObject(i); + result.add(new ScaffoldNameMappingRequestDTO(obj.getLong("scaffoldId"), obj.getString("name"))); + } + return result; + } + + public record ContigNameMappingRequestDTO(int contigId, String name) {} + + public record ScaffoldNameMappingRequestDTO(long scaffoldId, String name) {} +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/RenameContigRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/RenameContigRequestDTO.java new file mode 100644 index 0000000..a7a9458 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/RenameContigRequestDTO.java @@ -0,0 +1,38 @@ +/* + * MIT License + * + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package ru.itmo.ctlab.hict.hict_server.dto.request.names; + +import io.vertx.core.json.JsonObject; +import org.jetbrains.annotations.NotNull; +import org.jetbrains.annotations.Nullable; + +public record RenameContigRequestDTO(int contigId, @Nullable String newName) { + public static @NotNull RenameContigRequestDTO fromJSONObject(final @NotNull JsonObject jsonObject) { + return new RenameContigRequestDTO( + jsonObject.getInteger("contigId"), + jsonObject.getString("newName") + ); + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/RenameScaffoldRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/RenameScaffoldRequestDTO.java new file mode 100644 index 0000000..d10d37b --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/names/RenameScaffoldRequestDTO.java @@ -0,0 +1,38 @@ +/* + * MIT License + * + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package ru.itmo.ctlab.hict.hict_server.dto.request.names; + +import io.vertx.core.json.JsonObject; +import org.jetbrains.annotations.NotNull; +import org.jetbrains.annotations.Nullable; + +public record RenameScaffoldRequestDTO(long scaffoldId, @Nullable String newName) { + public static @NotNull RenameScaffoldRequestDTO fromJSONObject(final @NotNull JsonObject jsonObject) { + return new RenameScaffoldRequestDTO( + jsonObject.getLong("scaffoldId"), + jsonObject.getString("newName") + ); + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionRangeRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionRangeRequestDTO.java index 6c5c401..d8fff2c 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionRangeRequestDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionRangeRequestDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionToDebrisRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionToDebrisRequestDTO.java index a1f584b..b70e4ea 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionToDebrisRequestDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/MoveSelectionToDebrisRequestDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ReverseSelectionRangeRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ReverseSelectionRangeRequestDTO.java index 4ca5349..545bfdb 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ReverseSelectionRangeRequestDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ReverseSelectionRangeRequestDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ScaffoldRegionRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ScaffoldRegionRequestDTO.java index 81b7005..052af79 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ScaffoldRegionRequestDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/ScaffoldRegionRequestDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/SplitContigRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/SplitContigRequestDTO.java index c71e83c..0e184e9 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/SplitContigRequestDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/SplitContigRequestDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/UnscaffoldRegionRequestDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/UnscaffoldRegionRequestDTO.java index 356dd8d..59ecb3e 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/UnscaffoldRegionRequestDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/scaffolding/UnscaffoldRegionRequestDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/ContrastRangeSettingsDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/ContrastRangeSettingsDTO.java index 95a5c37..afe5550 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/ContrastRangeSettingsDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/ContrastRangeSettingsDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/NormalizationSettingsDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/NormalizationSettingsDTO.java index a00189d..f078bcf 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/NormalizationSettingsDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/request/tiles/NormalizationSettingsDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/AssemblyInfoDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/AssemblyInfoDTO.java index 0412b98..df17e25 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/AssemblyInfoDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/AssemblyInfoDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -37,7 +37,7 @@ public record AssemblyInfoDTO(@NotNull List<@NotNull ContigDescriptorDTO> contig final @NotNull var assemblyInfo = chunkedFile.getAssemblyInfo(); return new AssemblyInfoDTO( assemblyInfo.contigs().stream().map(ctg -> ContigDescriptorDTO.fromEntity(ctg, chunkedFile)).toList(), - assemblyInfo.scaffolds().stream().map(ScaffoldDescriptorDTO::fromEntity).toList() + assemblyInfo.scaffolds().stream().map(scaffold -> ScaffoldDescriptorDTO.fromEntity(scaffold, chunkedFile)).toList() ); } } diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/AssemblyInfoWithVersionDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/AssemblyInfoWithVersionDTO.java new file mode 100644 index 0000000..c1cef33 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/AssemblyInfoWithVersionDTO.java @@ -0,0 +1,30 @@ +/* + * MIT License + * + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package ru.itmo.ctlab.hict.hict_server.dto.response.assembly; + +import org.jetbrains.annotations.NotNull; + +public record AssemblyInfoWithVersionDTO(@NotNull AssemblyInfoDTO assemblyInfo, long version) { +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ContigDescriptorDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ContigDescriptorDTO.java index 207d23e..106be21 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ContigDescriptorDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ContigDescriptorDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -35,6 +35,7 @@ public record ContigDescriptorDTO( int contigId, String contigName, + String contigOriginalName, int contigDirection, long contigLengthBp, Map contigLengthBins, @@ -45,7 +46,8 @@ public record ContigDescriptorDTO( final var resolutions = chunkedFile.getResolutions(); return new ContigDescriptorDTO( ctg.descriptor().getContigId(), - ctg.descriptor().getContigName(), + chunkedFile.getContigDisplayName(ctg.descriptor().getContigId()), + chunkedFile.getContigOriginalName(ctg.descriptor().getContigId()), ctg.direction().ordinal(), ctg.descriptor().getLengthBp(), IntStream.range(1, resolutions.length).boxed().collect(Collectors.toMap(resIdx -> resolutions[resIdx], resIdx -> ctg.descriptor().getLengthBinsAtResolution()[resIdx])), diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ScaffoldDescriptorDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ScaffoldDescriptorDTO.java index 2b20a6c..cce3254 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ScaffoldDescriptorDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/assembly/ScaffoldDescriptorDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -32,14 +32,16 @@ public record ScaffoldDescriptorDTO( long scaffoldId, String scaffoldName, + String scaffoldOriginalName, long spacerLength, @Nullable ScaffoldDescriptor.ScaffoldBordersBP scaffoldBordersBP ) { - public static @NotNull ScaffoldDescriptorDTO fromEntity(final @NotNull ScaffoldTree.ScaffoldTuple scaffoldTuple) { + public static @NotNull ScaffoldDescriptorDTO fromEntity(final @NotNull ScaffoldTree.ScaffoldTuple scaffoldTuple, final @NotNull ru.itmo.ctlab.hict.hict_library.chunkedfile.ChunkedFile chunkedFile) { return new ScaffoldDescriptorDTO( scaffoldTuple.scaffoldDescriptor().scaffoldId(), - scaffoldTuple.scaffoldDescriptor().scaffoldName(), + chunkedFile.getScaffoldDisplayName(scaffoldTuple.scaffoldDescriptor().scaffoldId()), + chunkedFile.getScaffoldOriginalName(scaffoldTuple.scaffoldDescriptor().scaffoldId()), scaffoldTuple.scaffoldDescriptor().spacerLength(), scaffoldTuple.scaffoldBordersBP() ); diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/conversion/ConversionJobDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/conversion/ConversionJobDTO.java new file mode 100644 index 0000000..eaff772 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/conversion/ConversionJobDTO.java @@ -0,0 +1,25 @@ +package ru.itmo.ctlab.hict.hict_server.dto.response.conversion; + +import org.jetbrains.annotations.NotNull; + +import java.util.List; + +public record ConversionJobDTO( + @NotNull String jobId, + @NotNull String status, + @NotNull String sourceFilename, + @NotNull String outputFilename, + @NotNull String direction, + double overallProgress, + double resolutionProgress, + long currentResolution, + long elapsedMillis, + long etaMillis, + long resolutionElapsedMillis, + long resolutionEtaMillis, + long inputSizeBytes, + long outputSizeBytes, + @NotNull List<@NotNull String> logs, + @NotNull String error +) { +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/conversion/ConversionSubmitResponseDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/conversion/ConversionSubmitResponseDTO.java new file mode 100644 index 0000000..54bef9a --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/conversion/ConversionSubmitResponseDTO.java @@ -0,0 +1,9 @@ +package ru.itmo.ctlab.hict.hict_server.dto.response.conversion; + +import org.jetbrains.annotations.NotNull; + +public record ConversionSubmitResponseDTO( + @NotNull String status, + @NotNull String jobId +) { +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/fileop/OpenFileResponseDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/fileop/OpenFileResponseDTO.java index 4921c86..b061b4c 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/fileop/OpenFileResponseDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/fileop/OpenFileResponseDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/names/NameMappingDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/names/NameMappingDTO.java new file mode 100644 index 0000000..d36bbbc --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/response/names/NameMappingDTO.java @@ -0,0 +1,38 @@ +/* + * MIT License + * + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package ru.itmo.ctlab.hict.hict_server.dto.response.names; + +import org.jetbrains.annotations.NotNull; + +import java.util.List; + +public record NameMappingDTO( + @NotNull List<@NotNull ContigNameMappingDTO> contigs, + @NotNull List<@NotNull ScaffoldNameMappingDTO> scaffolds +) { + public record ContigNameMappingDTO(int contigId, @NotNull String originalName, @NotNull String name) {} + + public record ScaffoldNameMappingDTO(long scaffoldId, @NotNull String originalName, @NotNull String name) {} +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/ColormapDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/ColormapDTO.java index 7246305..4b29990 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/ColormapDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/ColormapDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/SimpleLinearGradientDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/SimpleLinearGradientDTO.java index a79fe72..56a308d 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/SimpleLinearGradientDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/SimpleLinearGradientDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/VisualizationOptionsDTO.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/VisualizationOptionsDTO.java index 537624c..0bb6eed 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/VisualizationOptionsDTO.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/dto/symmetric/visualization/VisualizationOptionsDTO.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/conversion/ConversionHandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/conversion/ConversionHandlersHolder.java new file mode 100644 index 0000000..f6fe702 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/conversion/ConversionHandlersHolder.java @@ -0,0 +1,485 @@ +package ru.itmo.ctlab.hict.hict_server.handlers.conversion; + +import io.vertx.core.Vertx; +import io.vertx.core.json.Json; +import io.vertx.ext.web.Router; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.jetbrains.annotations.NotNull; +import ru.itmo.ctlab.hict.hict_library.converters.ConversionOptions; +import ru.itmo.ctlab.hict.hict_library.converters.HictToMcoolConverter; +import ru.itmo.ctlab.hict.hict_library.converters.McoolToHictConverter; +import ru.itmo.ctlab.hict.hict_server.HandlersHolder; +import ru.itmo.ctlab.hict.hict_server.dto.response.conversion.ConversionJobDTO; +import ru.itmo.ctlab.hict.hict_server.dto.response.conversion.ConversionSubmitResponseDTO; +import ru.itmo.ctlab.hict.hict_server.util.shareable.ShareableWrappers; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.UUID; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +@RequiredArgsConstructor +@Slf4j +public class ConversionHandlersHolder extends HandlersHolder { + + private final Vertx vertx; + private static final long MAX_UPLOAD_BYTES = 2L * 1024 * 1024 * 1024; + private static final long JOB_TTL_MS = 60 * 60 * 1000; + + private static final Pattern OVERALL_PROGRESS_PATTERN = Pattern.compile( + "Overall progress: (\\d+)% \\((\\d+)/(\\d+)\\), elapsed=([0-9:]+), eta=([0-9:]+)" + ); + private static final Pattern RESOLUTION_PROGRESS_PATTERN = Pattern.compile( + "Resolution (\\d+) write: (\\d+)% \\((\\d+)/(\\d+) stripes\\), elapsed=([0-9:]+), eta=([0-9:]+)" + ); + + private final ConcurrentHashMap jobs = new ConcurrentHashMap<>(); + private final ConcurrentHashMap groups = new ConcurrentHashMap<>(); + + @Override + public void addHandlersToRouter(final @NotNull Router router) { + router.post("/convert/upload").blockingHandler(ctx -> { + try { + cleanupOldJobs(); + + final var upload = ctx.fileUploads().stream().findFirst().orElseThrow(() -> new IllegalArgumentException("No file uploaded")); + final var sourcePath = Path.of(upload.uploadedFileName()); + + if (Files.size(sourcePath) > MAX_UPLOAD_BYTES) { + Files.deleteIfExists(sourcePath); + throw new IllegalArgumentException("Uploaded file is too large"); + } + final var req = ctx.request(); + final var direction = req.getParam("direction"); + final var outputExt = "hict-to-mcool".equals(direction) ? ".mcool" : ".hict.hdf5"; + final var outputPath = Files.createTempFile("hict-converter-out-", outputExt); + + final var resolutionCsv = req.getParam("resolutions"); + final var resolutions = parseResolutions(resolutionCsv); + final var compression = parseInteger(req.getParam("compression"), 0); + final var compressionAlgorithm = ConversionOptions.CompressionAlgorithm.parse(req.getParam("compressionAlgorithm") == null ? "deflate" : req.getParam("compressionAlgorithm")); + final var chunkSize = parseInteger(req.getParam("chunkSize"), 8192); + final var applyAgpRaw = Boolean.parseBoolean(req.getParam("applyAgp")); + final var agpPathRaw = req.getParam("agpPath") == null ? ConversionOptions.NO_AGP : req.getParam("agpPath"); + final var parallelism = parseInteger(req.getParam("parallelism"), Runtime.getRuntime().availableProcessors()); + + final var useAgp = "hict-to-mcool".equals(direction) && applyAgpRaw; + final var agpPath = useAgp ? agpPathRaw : ConversionOptions.NO_AGP; + final var options = new ConversionOptions(sourcePath, outputPath, resolutions, chunkSize, compression, compressionAlgorithm, agpPath, useAgp, parallelism); + + final var job = createJob(sourcePath, outputPath, direction, parallelism, true, true); + submitJob(job, options, ensureGroup("upload", 1)); + + ctx.response().end(Json.encode(new ConversionSubmitResponseDTO("submitted", job.jobId))); + } catch (IOException e) { + throw new RuntimeException(e); + } + }); + + router.post("/convert/jobs").blockingHandler(ctx -> { + cleanupOldJobs(); + final var requestJson = ctx.body().asJsonObject(); + final var filename = requestJson.getString("filename"); + if (filename == null || filename.isBlank()) { + throw new IllegalArgumentException("filename is required"); + } + final var direction = requestJson.getString("direction", "mcool-to-hict"); + final var parallelism = requestJson.getInteger("parallelism", Runtime.getRuntime().availableProcessors()); + final var resolutions = parseResolutions(requestJson.getString("resolutions")); + final var compression = requestJson.getInteger("compression", 0); + final var compressionAlgorithm = ConversionOptions.CompressionAlgorithm.parse(requestJson.getString("compressionAlgorithm", "deflate")); + final var chunkSize = requestJson.getInteger("chunkSize", 8192); + + final var dataDirectoryWrapper = (ShareableWrappers.PathWrapper) vertx.sharedData().getLocalMap("hict_server").get("dataDirectory"); + if (dataDirectoryWrapper == null) { + throw new IllegalStateException("Data directory is not present in local map"); + } + final var dataDirectory = dataDirectoryWrapper.getPath(); + final var sourcePath = dataDirectory.resolve(filename).normalize(); + if (!sourcePath.startsWith(dataDirectory)) { + throw new IllegalArgumentException("Invalid filename"); + } + if (!Files.exists(sourcePath)) { + throw new IllegalArgumentException("Source file not found: " + filename); + } + + final var outputPath = deriveOutputPath(sourcePath); + if (Files.exists(outputPath)) { + throw new IllegalArgumentException("Output file already exists: " + outputPath.getFileName()); + } + + final var options = new ConversionOptions(sourcePath, outputPath, resolutions, chunkSize, compression, compressionAlgorithm, ConversionOptions.NO_AGP, false, parallelism); + final ConversionJob job; + try { + job = createJob(sourcePath, outputPath, direction, parallelism, false, false); + } catch (IOException e) { + throw new RuntimeException("Failed to create conversion job", e); + } + submitJob(job, options, ensureGroup(UUID.randomUUID().toString(), 1)); + + ctx.response().end(Json.encode(new ConversionSubmitResponseDTO("submitted", job.jobId))); + }); + + router.post("/convert/jobs/batch").blockingHandler(ctx -> { + cleanupOldJobs(); + final var requestJson = ctx.body().asJsonObject(); + final var files = requestJson.getJsonArray("files", null); + if (files == null || files.isEmpty()) { + throw new IllegalArgumentException("files is required"); + } + final var parallelJobs = Math.max(1, requestJson.getInteger("parallelJobs", 1)); + final var parallelism = requestJson.getInteger("parallelism", Runtime.getRuntime().availableProcessors()); + final var resolutions = parseResolutions(requestJson.getString("resolutions")); + final var compression = requestJson.getInteger("compression", 0); + final var compressionAlgorithm = ConversionOptions.CompressionAlgorithm.parse(requestJson.getString("compressionAlgorithm", "deflate")); + final var chunkSize = requestJson.getInteger("chunkSize", 8192); + + final var dataDirectoryWrapper = (ShareableWrappers.PathWrapper) vertx.sharedData().getLocalMap("hict_server").get("dataDirectory"); + if (dataDirectoryWrapper == null) { + throw new IllegalStateException("Data directory is not present in local map"); + } + final var dataDirectory = dataDirectoryWrapper.getPath(); + + final var groupId = UUID.randomUUID().toString(); + final var group = ensureGroup(groupId, parallelJobs); + final var jobIds = new ArrayList(); + + for (int i = 0; i < files.size(); i++) { + final var filename = files.getString(i); + final var sourcePath = dataDirectory.resolve(filename).normalize(); + if (!sourcePath.startsWith(dataDirectory)) { + throw new IllegalArgumentException("Invalid filename: " + filename); + } + if (!Files.exists(sourcePath)) { + throw new IllegalArgumentException("Source file not found: " + filename); + } + final var outputPath = deriveOutputPath(sourcePath); + if (Files.exists(outputPath)) { + throw new IllegalArgumentException("Output file already exists: " + outputPath.getFileName()); + } + final var options = new ConversionOptions(sourcePath, outputPath, resolutions, chunkSize, compression, compressionAlgorithm, ConversionOptions.NO_AGP, false, parallelism); + final ConversionJob job; + try { + job = createJob(sourcePath, outputPath, "mcool-to-hict", parallelism, false, false); + } catch (IOException e) { + throw new RuntimeException("Failed to create conversion job for " + filename, e); + } + submitJob(job, options, group); + jobIds.add(job.jobId); + } + + ctx.response().end(Json.encode(Map.of("status", "submitted", "groupId", groupId, "jobIds", jobIds))); + }); + + router.get("/convert/jobs").blockingHandler(ctx -> { + cleanupOldJobs(); + final var jobList = jobs.values().stream().map(ConversionJob::toDto).toList(); + ctx.response().end(Json.encode(jobList)); + }); + router.post("/convert/jobs/list").blockingHandler(ctx -> { + cleanupOldJobs(); + final var jobList = jobs.values().stream().map(ConversionJob::toDto).toList(); + ctx.response().end(Json.encode(jobList)); + }); + + router.get("/convert/jobs/:jobId").blockingHandler(ctx -> { + final var job = jobs.get(ctx.pathParam("jobId")); + if (job == null) { + throw new IllegalArgumentException("Job not found"); + } + ctx.response().end(Json.encode(job.toDto())); + }); + router.post("/convert/jobs/:jobId").blockingHandler(ctx -> { + final var job = jobs.get(ctx.pathParam("jobId")); + if (job == null) { + throw new IllegalArgumentException("Job not found"); + } + ctx.response().end(Json.encode(job.toDto())); + }); + + router.post("/convert/jobs/:jobId/stop").blockingHandler(ctx -> { + final var job = jobs.get(ctx.pathParam("jobId")); + if (job == null) { + throw new IllegalArgumentException("Job not found"); + } + job.requestCancel(); + ctx.response().end(Json.encode(Map.of("status", "cancelling", "jobId", job.jobId))); + }); + + router.get("/convert/download/:jobId").blockingHandler(ctx -> { + final var job = jobs.get(ctx.pathParam("jobId")); + if (job == null) { + throw new IllegalArgumentException("Job not found"); + } + if (!"finished".equals(job.status)) { + throw new IllegalStateException("Job is not finished yet"); + } + if (!Files.exists(job.outputPath)) { + throw new IllegalStateException("Converted file was already cleaned up"); + } + ctx.response().putHeader("Content-Type", "application/octet-stream"); + ctx.response().putHeader("Content-Disposition", "attachment; filename=\"" + job.outputPath.getFileName() + "\""); + ctx.response().sendFile(job.outputPath.toString()); + }); + } + + private static int parseInteger(final String value, final int defaultValue) { + if (value == null || value.isBlank()) { + return defaultValue; + } + return Integer.parseInt(value); + } + + private static @NotNull List parseResolutions(final String csv) { + if (csv == null || csv.isBlank()) { + return List.of(); + } + final var out = new ArrayList(); + for (final var token : csv.split(",")) { + final var trimmed = token.trim(); + if (!trimmed.isBlank()) { + out.add(Long.parseLong(trimmed)); + } + } + return out; + } + + private static Path deriveOutputPath(final @NotNull Path sourcePath) { + final var filename = sourcePath.getFileName().toString(); + final var lower = filename.toLowerCase(); + final String base; + if (lower.endsWith(".mcool")) { + base = filename.substring(0, filename.length() - ".mcool".length()); + } else if (lower.endsWith(".cool")) { + base = filename.substring(0, filename.length() - ".cool".length()); + } else { + base = filename; + } + return sourcePath.getParent().resolve(base + ".hict.hdf5"); + } + + private ConversionJob createJob(final @NotNull Path sourcePath, final @NotNull Path outputPath, final @NotNull String direction, final int parallelism, final boolean deleteSourceOnCleanup, final boolean deleteOutputOnCleanup) throws IOException { + final var jobId = UUID.randomUUID().toString(); + final var job = new ConversionJob(jobId, sourcePath, outputPath, direction, parallelism, deleteSourceOnCleanup, deleteOutputOnCleanup); + job.inputSizeBytes = Files.exists(sourcePath) ? Files.size(sourcePath) : 0L; + jobs.put(jobId, job); + return job; + } + + private ConversionJobGroup ensureGroup(final @NotNull String groupId, final int maxParallelJobs) { + return groups.computeIfAbsent(groupId, id -> new ConversionJobGroup(groupId, maxParallelJobs)); + } + + private void submitJob(final @NotNull ConversionJob job, final @NotNull ConversionOptions options, final @NotNull ConversionJobGroup group) { + group.totalJobs.incrementAndGet(); + group.executor.submit(() -> runJob(job, options, group)); + } + + private void runJob(final @NotNull ConversionJob job, final @NotNull ConversionOptions options, final @NotNull ConversionJobGroup group) { + job.status = "running"; + job.startedAtMs = Instant.now().toEpochMilli(); + final java.util.function.Consumer conversionLogger = message -> { + synchronized (System.out) { + System.out.println(message); + } + job.logs.add(message); + parseProgress(job, message); + job.updateOutputSize(); + }; + + try { + job.workerThread = Thread.currentThread(); + if ("hict-to-mcool".equals(job.direction)) { + new HictToMcoolConverter().convert(options, conversionLogger); + } else if ("mcool-to-hict".equals(job.direction)) { + new McoolToHictConverter().convert(options, conversionLogger); + } else { + throw new IllegalArgumentException("Unknown conversion direction"); + } + job.status = job.cancelRequested.get() ? "cancelled" : "finished"; + } catch (Exception e) { + if (job.cancelRequested.get()) { + job.status = "cancelled"; + job.error = "Cancelled"; + } else { + job.status = "failed"; + job.error = e.getMessage(); + job.logs.add("ERROR: " + e.getMessage()); + } + } finally { + job.finishedAtMs = Instant.now().toEpochMilli(); + group.onJobFinished(); + } + } + + private void parseProgress(final @NotNull ConversionJob job, final @NotNull String message) { + Matcher overall = OVERALL_PROGRESS_PATTERN.matcher(message); + if (overall.find()) { + job.overallProgress = clampPercent(overall.group(1)); + job.elapsedMillis = parseDurationMillis(overall.group(4)); + job.etaMillis = parseDurationMillis(overall.group(5)); + return; + } + Matcher res = RESOLUTION_PROGRESS_PATTERN.matcher(message); + if (res.find()) { + job.currentResolution = Long.parseLong(res.group(1)); + job.resolutionProgress = clampPercent(res.group(2)); + job.resolutionElapsedMillis = parseDurationMillis(res.group(5)); + job.resolutionEtaMillis = parseDurationMillis(res.group(6)); + } + } + + private static double clampPercent(final String value) { + try { + final var percent = Double.parseDouble(value); + return Math.max(0.0d, Math.min(100.0d, percent)) / 100.0d; + } catch (NumberFormatException ignored) { + return 0.0d; + } + } + + private static long parseDurationMillis(final String value) { + if (value == null || value.isBlank()) { + return 0L; + } + final var parts = value.split(":"); + if (parts.length == 2) { + final long minutes = Long.parseLong(parts[0]); + final long seconds = Long.parseLong(parts[1]); + return (minutes * 60 + seconds) * 1000L; + } + if (parts.length == 3) { + final long hours = Long.parseLong(parts[0]); + final long minutes = Long.parseLong(parts[1]); + final long seconds = Long.parseLong(parts[2]); + return (hours * 3600 + minutes * 60 + seconds) * 1000L; + } + return 0L; + } + + private void cleanupOldJobs() { + final var now = Instant.now().toEpochMilli(); + jobs.values().removeIf(job -> { + final var expired = now - job.createdAtMs > JOB_TTL_MS; + if (expired) { + try { + if (job.deleteSourceOnCleanup) { + Files.deleteIfExists(job.sourcePath); + } + if (job.deleteOutputOnCleanup) { + Files.deleteIfExists(job.outputPath); + } + } catch (IOException e) { + log.warn("Unable to cleanup temp files for {}", job.jobId, e); + } + } + return expired; + }); + } + + private static class ConversionJobGroup { + private final String groupId; + private final ExecutorService executor; + private final AtomicInteger totalJobs = new AtomicInteger(0); + private final AtomicInteger finishedJobs = new AtomicInteger(0); + + private ConversionJobGroup(final String groupId, final int maxParallelJobs) { + this.groupId = groupId; + this.executor = Executors.newFixedThreadPool(maxParallelJobs); + } + + private void onJobFinished() { + if (finishedJobs.incrementAndGet() >= totalJobs.get()) { + executor.shutdown(); + } + } + } + + private static class ConversionJob { + private final String jobId; + private final long createdAtMs = Instant.now().toEpochMilli(); + private final Path sourcePath; + private final Path outputPath; + private final String direction; + private final int parallelism; + private final boolean deleteSourceOnCleanup; + private final boolean deleteOutputOnCleanup; + private volatile String status = "queued"; + private volatile String error = ""; + private final CopyOnWriteArrayList logs = new CopyOnWriteArrayList<>(); + private volatile double overallProgress = 0.0d; + private volatile double resolutionProgress = 0.0d; + private volatile long currentResolution = 0L; + private volatile long elapsedMillis = 0L; + private volatile long etaMillis = 0L; + private volatile long resolutionElapsedMillis = 0L; + private volatile long resolutionEtaMillis = 0L; + private volatile long inputSizeBytes = 0L; + private volatile long outputSizeBytes = 0L; + private volatile long startedAtMs = 0L; + private volatile long finishedAtMs = 0L; + private volatile Thread workerThread = null; + private final AtomicBoolean cancelRequested = new AtomicBoolean(false); + + private ConversionJob(String jobId, Path sourcePath, Path outputPath, String direction, int parallelism, boolean deleteSourceOnCleanup, boolean deleteOutputOnCleanup) { + this.jobId = jobId; + this.sourcePath = sourcePath; + this.outputPath = outputPath; + this.direction = direction; + this.parallelism = parallelism; + this.deleteSourceOnCleanup = deleteSourceOnCleanup; + this.deleteOutputOnCleanup = deleteOutputOnCleanup; + } + + private void updateOutputSize() { + try { + if (Files.exists(outputPath)) { + outputSizeBytes = Files.size(outputPath); + } + } catch (IOException ignored) { + // ignore + } + } + + private void requestCancel() { + cancelRequested.set(true); + if (workerThread != null) { + workerThread.interrupt(); + } + } + + private ConversionJobDTO toDto() { + return new ConversionJobDTO( + jobId, + status, + sourcePath.getFileName().toString(), + outputPath.getFileName().toString(), + direction, + overallProgress, + resolutionProgress, + currentResolution, + elapsedMillis, + etaMillis, + resolutionElapsedMillis, + resolutionEtaMillis, + inputSizeBytes, + outputSizeBytes, + List.copyOf(logs), + error == null ? "" : error + ); + } + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/fileop/FileOpHandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/fileop/FileOpHandlersHolder.java index 644b127..7496517 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/fileop/FileOpHandlersHolder.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/fileop/FileOpHandlersHolder.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -27,7 +27,9 @@ import io.vertx.core.Vertx; import io.vertx.core.buffer.Buffer; import io.vertx.core.json.Json; +import io.vertx.core.shareddata.LocalMap; import io.vertx.ext.web.Router; +import lombok.NonNull; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.ArrayUtils; @@ -75,7 +77,7 @@ public void addHandlersToRouter(final @NotNull Router router) { return; } - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); final var chunkedFile = new ChunkedFile( new ChunkedFile.ChunkedFileOptions( @@ -95,7 +97,7 @@ public void addHandlersToRouter(final @NotNull Router router) { }); router.post("/get_agp_for_assembly").blockingHandler(ctx -> { - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); if (chunkedFileWrapper == null) { @@ -118,7 +120,7 @@ public void addHandlersToRouter(final @NotNull Router router) { }); router.post("/load_agp").blockingHandler(ctx -> { - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); if (chunkedFileWrapper == null) { diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/files/FSHandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/files/FSHandlersHolder.java index 7d150ed..fde5a9f 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/files/FSHandlersHolder.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/files/FSHandlersHolder.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/info/InfoHandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/info/InfoHandlersHolder.java new file mode 100644 index 0000000..7b7c7f2 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/info/InfoHandlersHolder.java @@ -0,0 +1,85 @@ +package ru.itmo.ctlab.hict.hict_server.handlers.info; + +import io.vertx.core.Vertx; +import io.vertx.core.json.Json; +import io.vertx.ext.web.Router; +import org.jetbrains.annotations.NotNull; +import ru.itmo.ctlab.hict.hict_server.HandlersHolder; + +import java.io.BufferedReader; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; + +public class InfoHandlersHolder extends HandlersHolder { + private final @NotNull Vertx vertx; + + public InfoHandlersHolder(final @NotNull Vertx vertx) { + this.vertx = vertx; + } + + @Override + public void addHandlersToRouter(final @NotNull Router router) { + router.get("/version").handler(ctx -> { + final var version = readVersion(); + final var webuiVersion = readWebUiVersion(); + ctx.response().setStatusCode(200).end(Json.encode(Map.of( + "version", version, + "webuiVersion", webuiVersion + ))); + }); + } + + private @NotNull String readVersion() { + final var systemProp = System.getProperty("hict.version"); + if (systemProp != null && !systemProp.isBlank()) { + return systemProp.trim(); + } + try { + final var versionPath = Path.of("version.txt"); + if (Files.exists(versionPath)) { + return Files.readString(versionPath).trim(); + } + } catch (final Exception ignored) { + // ignore + } + try (final InputStream stream = getClass().getResourceAsStream("/version.txt")) { + if (stream != null) { + try (final BufferedReader reader = + new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))) { + final var line = reader.readLine(); + if (line != null && !line.isBlank()) { + return line.trim(); + } + } + } + } catch (final Exception ignored) { + // ignore + } + return "unknown"; + } + + private @NotNull String readWebUiVersion() { + try (final InputStream stream = getClass().getResourceAsStream("/webui-package.json")) { + if (stream != null) { + final var json = new String(stream.readAllBytes(), StandardCharsets.UTF_8); + final var marker = "\"version\""; + final var idx = json.indexOf(marker); + if (idx >= 0) { + final var colon = json.indexOf(':', idx); + final var quoteStart = json.indexOf('"', colon + 1); + final var quoteEnd = json.indexOf('"', quoteStart + 1); + if (quoteStart >= 0 && quoteEnd > quoteStart) { + return json.substring(quoteStart + 1, quoteEnd); + } + } + } + } catch (final Exception ignored) { + // ignore + } + return "unknown"; + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/names/NameMappingHandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/names/NameMappingHandlersHolder.java new file mode 100644 index 0000000..06526e0 --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/names/NameMappingHandlersHolder.java @@ -0,0 +1,207 @@ +/* + * MIT License + * + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +package ru.itmo.ctlab.hict.hict_server.handlers.names; + +import io.vertx.core.Vertx; +import io.vertx.core.json.Json; +import io.vertx.core.shareddata.LocalMap; +import io.vertx.ext.web.Router; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.jetbrains.annotations.NotNull; +import ru.itmo.ctlab.hict.hict_library.chunkedfile.ChunkedFile; +import ru.itmo.ctlab.hict.hict_server.HandlersHolder; +import ru.itmo.ctlab.hict.hict_server.dto.request.names.ImportNameMappingRequestDTO; +import ru.itmo.ctlab.hict.hict_server.dto.request.names.RenameContigRequestDTO; +import ru.itmo.ctlab.hict.hict_server.dto.request.names.RenameScaffoldRequestDTO; +import ru.itmo.ctlab.hict.hict_server.dto.response.assembly.AssemblyInfoDTO; +import ru.itmo.ctlab.hict.hict_server.dto.response.assembly.AssemblyInfoWithVersionDTO; +import ru.itmo.ctlab.hict.hict_server.dto.response.names.NameMappingDTO; +import ru.itmo.ctlab.hict.hict_server.handlers.util.TileStatisticHolder; +import ru.itmo.ctlab.hict.hict_server.util.shareable.ShareableWrappers; + +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; + +@RequiredArgsConstructor +@Slf4j +public class NameMappingHandlersHolder extends HandlersHolder { + private final Vertx vertx; + + @Override + public void addHandlersToRouter(final @NotNull Router router) { + router.post("/names/contig").blockingHandler(ctx -> { + final var request = RenameContigRequestDTO.fromJSONObject(ctx.body().asJsonObject()); + final var chunkedFile = extractChunkedFile(ctx); + if (chunkedFile == null) { + return; + } + + final var newName = normalizeName(request.newName()); + validateContigRename(chunkedFile, request.contigId(), newName); + chunkedFile.setContigNameOverride(request.contigId(), newName); + + final var newVersion = incrementVersionAndResetTileStats(chunkedFile); + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); + }); + + router.post("/names/scaffold").blockingHandler(ctx -> { + final var request = RenameScaffoldRequestDTO.fromJSONObject(ctx.body().asJsonObject()); + final var chunkedFile = extractChunkedFile(ctx); + if (chunkedFile == null) { + return; + } + + final var newName = normalizeName(request.newName()); + validateScaffoldRename(chunkedFile, request.scaffoldId(), newName); + chunkedFile.setScaffoldNameOverride(request.scaffoldId(), newName); + + final var newVersion = incrementVersionAndResetTileStats(chunkedFile); + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); + }); + + router.get("/names/export").blockingHandler(ctx -> { + final var chunkedFile = extractChunkedFile(ctx); + if (chunkedFile == null) { + return; + } + + final var contigs = chunkedFile.getContigTree().getOrderedContigList().stream().map(tuple -> + new NameMappingDTO.ContigNameMappingDTO( + tuple.descriptor().getContigId(), + chunkedFile.getContigOriginalName(tuple.descriptor().getContigId()), + chunkedFile.getContigDisplayName(tuple.descriptor().getContigId()) + ) + ).toList(); + + final var scaffolds = chunkedFile.getScaffoldTree().getScaffoldList().stream().map(tuple -> + new NameMappingDTO.ScaffoldNameMappingDTO( + tuple.scaffoldDescriptor().scaffoldId(), + chunkedFile.getScaffoldOriginalName(tuple.scaffoldDescriptor().scaffoldId()), + chunkedFile.getScaffoldDisplayName(tuple.scaffoldDescriptor().scaffoldId()) + ) + ).toList(); + + ctx.response().end(Json.encode(new NameMappingDTO(contigs, scaffolds))); + }); + + router.post("/names/import").blockingHandler(ctx -> { + final var request = ImportNameMappingRequestDTO.fromJSONObject(ctx.body().asJsonObject()); + final var chunkedFile = extractChunkedFile(ctx); + if (chunkedFile == null) { + return; + } + + final Map contigUpdates = new HashMap<>(); + request.contigs().forEach(entry -> contigUpdates.put(entry.contigId(), normalizeName(entry.name()))); + final Map scaffoldUpdates = new HashMap<>(); + request.scaffolds().forEach(entry -> scaffoldUpdates.put(entry.scaffoldId(), normalizeName(entry.name()))); + + validateContigMappingImport(chunkedFile, contigUpdates); + validateScaffoldMappingImport(chunkedFile, scaffoldUpdates); + + contigUpdates.forEach(chunkedFile::setContigNameOverride); + scaffoldUpdates.forEach(chunkedFile::setScaffoldNameOverride); + + final var newVersion = incrementVersionAndResetTileStats(chunkedFile); + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); + }); + } + + private ChunkedFile extractChunkedFile(final @NotNull io.vertx.ext.web.RoutingContext ctx) { + final @NotNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); + final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); + if (chunkedFileWrapper == null) { + ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + return null; + } + return chunkedFileWrapper.getChunkedFile(); + } + + private long incrementVersionAndResetTileStats(final @NotNull ChunkedFile chunkedFile) { + final @NotNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); + final var stats = (TileStatisticHolder) map.get("TileStatisticHolder"); + if (stats == null) { + throw new RuntimeException("Tile statistics is not present in the local map, maybe the file is not yet opened?"); + } + final var newStats = TileStatisticHolder.resetRangesWithIncrementedVersion(stats, chunkedFile.getResolutions().length); + map.put("TileStatisticHolder", newStats); + return newStats.versionCounter().get(); + } + + private static @NotNull String normalizeName(final String name) { + if (name == null) { + return ""; + } + return name.trim(); + } + + private void validateContigRename(final @NotNull ChunkedFile chunkedFile, final int contigId, final @NotNull String newName) { + if (newName.isBlank()) { + return; + } + final var existingNames = new HashSet<>(chunkedFile.getAllContigDisplayNames()); + existingNames.remove(chunkedFile.getContigDisplayName(contigId)); + if (existingNames.contains(newName)) { + throw new IllegalArgumentException("Contig name must be unique. Name '" + newName + "' already exists."); + } + } + + private void validateScaffoldRename(final @NotNull ChunkedFile chunkedFile, final long scaffoldId, final @NotNull String newName) { + if (newName.isBlank()) { + return; + } + final var existingNames = new HashSet<>(chunkedFile.getAllScaffoldDisplayNames()); + existingNames.remove(chunkedFile.getScaffoldDisplayName(scaffoldId)); + if (existingNames.contains(newName)) { + throw new IllegalArgumentException("Scaffold name must be unique. Name '" + newName + "' already exists."); + } + } + + private void validateContigMappingImport(final @NotNull ChunkedFile chunkedFile, final @NotNull Map updates) { + final var finalNames = new HashMap(); + chunkedFile.getContigTree().getContigDescriptors().keySet().forEach(id -> finalNames.put(id, chunkedFile.getContigDisplayName(id))); + updates.forEach((id, name) -> finalNames.put(id, name.isBlank() ? chunkedFile.getContigOriginalName(id) : name)); + final var seen = new HashSet(); + for (final var name : finalNames.values()) { + if (!seen.add(name)) { + throw new IllegalArgumentException("Contig name must be unique. Duplicate name '" + name + "' in import."); + } + } + } + + private void validateScaffoldMappingImport(final @NotNull ChunkedFile chunkedFile, final @NotNull Map updates) { + final var finalNames = new HashMap(); + chunkedFile.getScaffoldTree().getScaffoldList().forEach(tuple -> finalNames.put(tuple.scaffoldDescriptor().scaffoldId(), chunkedFile.getScaffoldDisplayName(tuple.scaffoldDescriptor().scaffoldId()))); + updates.forEach((id, name) -> finalNames.put(id, name.isBlank() ? chunkedFile.getScaffoldOriginalName(id) : name)); + final var seen = new HashSet(); + for (final var name : finalNames.values()) { + if (!seen.add(name)) { + throw new IllegalArgumentException("Scaffold name must be unique. Duplicate name '" + name + "' in import."); + } + } + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/operations/ScaffoldingOpHandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/operations/ScaffoldingOpHandlersHolder.java index f3d842c..1ba8e9b 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/operations/ScaffoldingOpHandlersHolder.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/operations/ScaffoldingOpHandlersHolder.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -26,16 +26,20 @@ import io.vertx.core.Vertx; import io.vertx.core.json.Json; +import io.vertx.core.shareddata.LocalMap; import io.vertx.ext.web.Router; import lombok.NonNull; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.jetbrains.annotations.NotNull; +import ru.itmo.ctlab.hict.hict_library.chunkedfile.ChunkedFile; import ru.itmo.ctlab.hict.hict_library.chunkedfile.resolution.ResolutionDescriptor; import ru.itmo.ctlab.hict.hict_library.domain.QueryLengthUnit; import ru.itmo.ctlab.hict.hict_server.HandlersHolder; import ru.itmo.ctlab.hict.hict_server.dto.request.scaffolding.*; import ru.itmo.ctlab.hict.hict_server.dto.response.assembly.AssemblyInfoDTO; +import ru.itmo.ctlab.hict.hict_server.dto.response.assembly.AssemblyInfoWithVersionDTO; +import ru.itmo.ctlab.hict.hict_server.handlers.util.TileStatisticHolder; import ru.itmo.ctlab.hict.hict_server.util.shareable.ShareableWrappers; @RequiredArgsConstructor @@ -51,19 +55,21 @@ public void addHandlersToRouter(final @NotNull Router router) { final @NotNull @NonNull var request = ReverseSelectionRangeRequestDTO.fromJSONObject(requestJSON); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); - final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); - if (chunkedFileWrapper == null) { - ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + final var chunkedFile = extractChunkedFile(map, ctx); + if (chunkedFile == null) { return; } - final var chunkedFile = chunkedFileWrapper.getChunkedFile(); log.debug("Got ChunkedFile from map"); chunkedFile.scaffoldingOperations().reverseSelectionRangeBp(request.startBP(), request.endBP()); - ctx.response().end(Json.encode(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile))); + final var newVersion = incrementVersionAndResetTileStats(map, chunkedFile, ctx); + if (newVersion == null) { + return; + } + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); }); router.post("/move_selection_range").blockingHandler(ctx -> { final @NotNull var requestBody = ctx.body(); @@ -71,19 +77,21 @@ public void addHandlersToRouter(final @NotNull Router router) { final @NotNull @NonNull var request = MoveSelectionRangeRequestDTO.fromJSONObject(requestJSON); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); - final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); - if (chunkedFileWrapper == null) { - ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + final var chunkedFile = extractChunkedFile(map, ctx); + if (chunkedFile == null) { return; } - final var chunkedFile = chunkedFileWrapper.getChunkedFile(); log.debug("Got ChunkedFile from map"); chunkedFile.scaffoldingOperations().moveSelectionRangeBp(request.startBP(), request.endBP(), request.targetStartBP()); - ctx.response().end(Json.encode(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile))); + final var newVersion = incrementVersionAndResetTileStats(map, chunkedFile, ctx); + if (newVersion == null) { + return; + } + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); }); router.post("/split_contig_at_bin").blockingHandler(ctx -> { final @NotNull var requestBody = ctx.body(); @@ -91,19 +99,21 @@ public void addHandlersToRouter(final @NotNull Router router) { final @NotNull @NonNull var request = SplitContigRequestDTO.fromJSONObject(requestJSON); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); - final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); - if (chunkedFileWrapper == null) { - ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + final var chunkedFile = extractChunkedFile(map, ctx); + if (chunkedFile == null) { return; } - final var chunkedFile = chunkedFileWrapper.getChunkedFile(); log.debug("Got ChunkedFile from map"); chunkedFile.scaffoldingOperations().splitContigAtBin(request.splitPx(), ResolutionDescriptor.fromBpResolution(request.bpResolution(), chunkedFile), QueryLengthUnit.PIXELS); - ctx.response().end(Json.encode(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile))); + final var newVersion = incrementVersionAndResetTileStats(map, chunkedFile, ctx); + if (newVersion == null) { + return; + } + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); }); router.post("/group_contigs_into_scaffold").blockingHandler(ctx -> { final @NotNull var requestBody = ctx.body(); @@ -111,19 +121,21 @@ public void addHandlersToRouter(final @NotNull Router router) { final @NotNull @NonNull var request = ScaffoldRegionRequestDTO.fromJSONObject(requestJSON); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); - final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); - if (chunkedFileWrapper == null) { - ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + final var chunkedFile = extractChunkedFile(map, ctx); + if (chunkedFile == null) { return; } - final var chunkedFile = chunkedFileWrapper.getChunkedFile(); log.debug("Got ChunkedFile from map"); chunkedFile.scaffoldingOperations().scaffoldRegion(request.startBP(), request.endBP(), ResolutionDescriptor.fromResolutionOrder(0), QueryLengthUnit.BASE_PAIRS, null); - ctx.response().end(Json.encode(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile))); + final var newVersion = incrementVersionAndResetTileStats(map, chunkedFile, ctx); + if (newVersion == null) { + return; + } + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); }); router.post("/ungroup_contigs_from_scaffold").blockingHandler(ctx -> { final @NotNull var requestBody = ctx.body(); @@ -131,19 +143,21 @@ public void addHandlersToRouter(final @NotNull Router router) { final @NotNull @NonNull var request = UnscaffoldRegionRequestDTO.fromJSONObject(requestJSON); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); - final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); - if (chunkedFileWrapper == null) { - ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + final var chunkedFile = extractChunkedFile(map, ctx); + if (chunkedFile == null) { return; } - final var chunkedFile = chunkedFileWrapper.getChunkedFile(); log.debug("Got ChunkedFile from map"); chunkedFile.scaffoldingOperations().unscaffoldRegion(request.startBP(), request.endBP(), ResolutionDescriptor.fromResolutionOrder(0), QueryLengthUnit.BASE_PAIRS); - ctx.response().end(Json.encode(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile))); + final var newVersion = incrementVersionAndResetTileStats(map, chunkedFile, ctx); + if (newVersion == null) { + return; + } + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); }); router.post("/move_selection_to_debris").blockingHandler(ctx -> { final @NotNull var requestBody = ctx.body(); @@ -151,19 +165,43 @@ public void addHandlersToRouter(final @NotNull Router router) { final @NotNull @NonNull var request = MoveSelectionToDebrisRequestDTO.fromJSONObject(requestJSON); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); - final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); - if (chunkedFileWrapper == null) { - ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + final var chunkedFile = extractChunkedFile(map, ctx); + if (chunkedFile == null) { return; } - final var chunkedFile = chunkedFileWrapper.getChunkedFile(); log.debug("Got ChunkedFile from map"); chunkedFile.scaffoldingOperations().moveRegionToDebris(request.startBP(), request.endBP(), ResolutionDescriptor.fromResolutionOrder(0), QueryLengthUnit.BASE_PAIRS); - ctx.response().end(Json.encode(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile))); + final var newVersion = incrementVersionAndResetTileStats(map, chunkedFile, ctx); + if (newVersion == null) { + return; + } + ctx.response().end(Json.encode(new AssemblyInfoWithVersionDTO(AssemblyInfoDTO.generateFromChunkedFile(chunkedFile), newVersion))); }); } + + private ChunkedFile extractChunkedFile(final @NotNull LocalMap map, final @NotNull io.vertx.ext.web.RoutingContext ctx) { + final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); + if (chunkedFileWrapper == null) { + ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + return null; + } + return chunkedFileWrapper.getChunkedFile(); + } + + private Long incrementVersionAndResetTileStats(final @NotNull LocalMap map, + final @NotNull ChunkedFile chunkedFile, + final @NotNull io.vertx.ext.web.RoutingContext ctx) { + final var stats = (TileStatisticHolder) map.get("TileStatisticHolder"); + if (stats == null) { + ctx.fail(new RuntimeException("Tile statistics is not present in the local map, maybe the file is not yet opened?")); + return null; + } + final var newStats = TileStatisticHolder.resetRangesWithIncrementedVersion(stats, chunkedFile.getResolutions().length); + map.put("TileStatisticHolder", newStats); + return newStats.versionCounter().get(); + } } diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/tiles/TileHandlersHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/tiles/TileHandlersHolder.java index ac25b81..fa9d343 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/tiles/TileHandlersHolder.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/tiles/TileHandlersHolder.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -27,6 +27,7 @@ import io.vertx.core.Vertx; import io.vertx.core.buffer.Buffer; import io.vertx.core.json.Json; +import io.vertx.core.shareddata.LocalMap; import io.vertx.ext.web.Router; import lombok.NonNull; import lombok.RequiredArgsConstructor; @@ -59,7 +60,7 @@ public void addHandlersToRouter(final @NotNull Router router) { final @NotNull @NonNull var request = VisualizationOptionsDTO.fromJSONObject(requestJSON); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); map.put("visualizationOptions", new ShareableWrappers.SimpleVisualizationOptionsWrapper(request.toEntity())); final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); @@ -75,9 +76,7 @@ public void addHandlersToRouter(final @NotNull Router router) { ctx.fail(new RuntimeException("Tile statistics is not present in the local map, maybe the file is not yet opened?")); return; } - final var newStats = TileStatisticHolder.newDefaultStatisticHolder(chunkedFile.getResolutions().length); - newStats.versionCounter().set(stats.versionCounter().get()); - map.put("TileStatisticHolder", newStats); + map.put("TileStatisticHolder", TileStatisticHolder.resetRangesKeepingVersion(stats, chunkedFile.getResolutions().length)); final var visualizationOptionsWrapper = ((ShareableWrappers.SimpleVisualizationOptionsWrapper) (map.get("visualizationOptions"))); if (visualizationOptionsWrapper == null) { ctx.fail(new RuntimeException("Visualization options are not present in the local map, maybe the file is not yet opened?")); @@ -88,7 +87,7 @@ public void addHandlersToRouter(final @NotNull Router router) { }); router.post("/get_visualization_options").blockingHandler(ctx -> { - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); if (chunkedFileWrapper == null) { @@ -106,6 +105,24 @@ public void addHandlersToRouter(final @NotNull Router router) { ctx.response().setStatusCode(200).end(Json.encode(VisualizationOptionsDTO.fromEntity(options, chunkedFile))); }); + router.post("/tiles/reload").blockingHandler(ctx -> { + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); + final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); + if (chunkedFileWrapper == null) { + ctx.fail(new RuntimeException("Chunked file is not present in the local map, maybe the file is not yet opened?")); + return; + } + final var chunkedFile = chunkedFileWrapper.getChunkedFile(); + final var stats = (TileStatisticHolder) map.get("TileStatisticHolder"); + if (stats == null) { + ctx.fail(new RuntimeException("Tile statistics is not present in the local map, maybe the file is not yet opened?")); + return; + } + final var newStats = TileStatisticHolder.resetRangesWithIncrementedVersion(stats, chunkedFile.getResolutions().length); + map.put("TileStatisticHolder", newStats); + ctx.response().setStatusCode(200).end(Json.encode(Map.of("version", newStats.versionCounter().get()))); + }); + router.get("/get_tile").handler(ctx -> { log.debug("Entered non-blocking handler"); ctx.next(); @@ -114,14 +131,14 @@ public void addHandlersToRouter(final @NotNull Router router) { final var row = Long.parseLong(ctx.request().getParam("row", "0")); final var col = Long.parseLong(ctx.request().getParam("col", "0")); - final var version = Long.parseLong(ctx.request().getParam("version", "0")); + final var requestedVersion = Long.parseLong(ctx.request().getParam("version", "0")); final int tileHeight; final int tileWidth; final var format = TileFormat.valueOf(ctx.request().getParam("format", "JSON_PNG_WITH_RANGES")); log.debug("Got parameters"); - final var map = vertx.sharedData().getLocalMap("hict_server"); + final @NotNull @NonNull LocalMap map = vertx.sharedData().getLocalMap("hict_server"); log.debug("Got map"); final var chunkedFileWrapper = ((ShareableWrappers.ChunkedFileWrapper) (map.get("chunkedFile"))); @@ -138,7 +155,19 @@ public void addHandlersToRouter(final @NotNull Router router) { } final var options = visualizationOptionsWrapper.getSimpleVisualizationOptions(); - final var level = chunkedFile.getResolutions().length - Integer.parseInt(ctx.request().getParam("level", "0")); + final var requestedBpResolutionParam = ctx.request().getParam("bpResolution"); + final int level; + if (requestedBpResolutionParam != null) { + final var requestedBpResolution = Long.parseLong(requestedBpResolutionParam); + final var resolutionOrder = chunkedFile.getResolutionToIndex().get(requestedBpResolution); + if (resolutionOrder == null) { + ctx.fail(new RuntimeException("Requested bpResolution is not present in opened file: " + requestedBpResolution)); + return; + } + level = resolutionOrder; + } else { + level = chunkedFile.getResolutions().length - Integer.parseInt(ctx.request().getParam("level", "0")); + } final var stats = (TileStatisticHolder) map.get("TileStatisticHolder"); if (stats == null) { @@ -147,10 +176,10 @@ public void addHandlersToRouter(final @NotNull Router router) { } var currentVersion = stats.versionCounter().get(); + long version = requestedVersion; if (version < currentVersion) { - log.debug(String.format("Current version is %d and request version is %d", currentVersion, version)); - ctx.response().setStatusCode(204).putHeader("Content-Type", "text/plain").end(String.format("Current version is %d and request version is %d", currentVersion, version)); - return; + log.debug(String.format("Current version is %d and request version is %d; serving with current version", currentVersion, version)); + version = currentVersion; } do { currentVersion = stats.versionCounter().get(); diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/util/TileStatisticHolder.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/util/TileStatisticHolder.java index ebc87d0..d2f8022 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/util/TileStatisticHolder.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/handlers/util/TileStatisticHolder.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -43,4 +43,20 @@ public static TileStatisticHolder newDefaultStatisticHolder(final int resolution ); } + public static @NotNull TileStatisticHolder newDefaultStatisticHolderWithVersion(final int resolutionCount, final long version) { + final var stats = newDefaultStatisticHolder(resolutionCount); + stats.versionCounter().set(version); + return stats; + } + + public static @NotNull TileStatisticHolder resetRangesKeepingVersion(final @NotNull TileStatisticHolder previous, + final int resolutionCount) { + return newDefaultStatisticHolderWithVersion(resolutionCount, previous.versionCounter().get()); + } + + public static @NotNull TileStatisticHolder resetRangesWithIncrementedVersion(final @NotNull TileStatisticHolder previous, + final int resolutionCount) { + return newDefaultStatisticHolderWithVersion(resolutionCount, previous.versionCounter().incrementAndGet()); + } + } diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/tools/ConversionCliLauncher.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/tools/ConversionCliLauncher.java new file mode 100644 index 0000000..c14056f --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/tools/ConversionCliLauncher.java @@ -0,0 +1,93 @@ +package ru.itmo.ctlab.hict.hict_server.tools; + +import ru.itmo.ctlab.hict.hict_library.chunkedfile.hdf5.HDF5LibraryInitializer; +import ru.itmo.ctlab.hict.hict_library.converters.ConversionOptions; +import ru.itmo.ctlab.hict.hict_library.converters.HictToMcoolConverter; +import ru.itmo.ctlab.hict.hict_library.converters.McoolToHictConverter; + +import java.nio.file.Path; +import java.util.Arrays; +import java.util.List; +import java.util.function.Consumer; + +public class ConversionCliLauncher { + + static { + HDF5LibraryInitializer.initializeHDF5Library(); + } + + public static void main(String[] args) throws Exception { + if (args.length == 0 || "help".equals(args[0])) { + printHelp(); + return; + } + + final var command = args[0]; + final var parser = new ArgParser(Arrays.copyOfRange(args, 1, args.length)); + final var options = new ConversionOptions( + Path.of(parser.require("input")), + Path.of(parser.require("output")), + parser.listOfLong("resolutions"), + parser.integer("chunk-size", 8192), + parser.integer("compression", 0), + ConversionOptions.CompressionAlgorithm.parse(parser.value("compression-algorithm", "deflate")), + parser.value("agp", ConversionOptions.NO_AGP), + parser.flag("apply-agp"), + parser.integer("parallelism", Runtime.getRuntime().availableProcessors()) + ); + + switch (command) { + case "hict-to-mcool" -> new HictToMcoolConverter().convert(options, stdoutLogger()); + case "mcool-to-hict" -> new McoolToHictConverter().convert(options, stdoutLogger()); + default -> throw new IllegalArgumentException("Unknown command: " + command); + } + } + + private static Consumer stdoutLogger() { + return message -> { + synchronized (System.out) { + System.out.println(message); + System.out.flush(); + } + }; + } + + private static void printHelp() { + System.out.println("Usage:"); + System.out.println(" hict-to-mcool --input= --output= [--resolutions=10000,50000] [--compression=0..9] [--compression-algorithm=deflate|zstd|lzf] [--chunk-size=8192] [--agp=foo.agp --apply-agp] [--parallelism=N]"); + System.out.println(" mcool-to-hict --input= --output= [--resolutions=10000,50000] [--compression=0..9] [--compression-algorithm=deflate|zstd|lzf] [--chunk-size=8192] [--parallelism=N]"); + } + + private record ArgParser(String[] args) { + String require(String key) { + final var value = value(key, null); + if (value == null || value.isBlank()) { + throw new IllegalArgumentException("Missing required --" + key); + } + return value; + } + + String value(String key, String defaultValue) { + final var prefix = "--" + key + "="; + return Arrays.stream(args).filter(a -> a.startsWith(prefix)).map(a -> a.substring(prefix.length())).findFirst().orElse(defaultValue); + } + + int integer(String key, int defaultValue) { + final var value = value(key, String.valueOf(defaultValue)); + return Integer.parseInt(value); + } + + boolean flag(String key) { + final var prefix = "--" + key; + return Arrays.stream(args).anyMatch(prefix::equals); + } + + List listOfLong(String key) { + final var value = value(key, ""); + if (value.isBlank()) { + return List.of(); + } + return Arrays.stream(value.split(",")).map(String::trim).filter(s -> !s.isBlank()).map(Long::parseLong).toList(); + } + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/tools/HictCli.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/tools/HictCli.java new file mode 100644 index 0000000..ca0c8af --- /dev/null +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/tools/HictCli.java @@ -0,0 +1,204 @@ +package ru.itmo.ctlab.hict.hict_server.tools; + +import io.vertx.core.Launcher; +import picocli.CommandLine; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; +import picocli.CommandLine.Parameters; +import ru.itmo.ctlab.hict.hict_library.chunkedfile.hdf5.HDF5LibraryInitializer; +import ru.itmo.ctlab.hict.hict_library.converters.ConversionOptions; +import ru.itmo.ctlab.hict.hict_library.converters.HictToMcoolConverter; +import ru.itmo.ctlab.hict.hict_library.converters.McoolToHictConverter; +import ru.itmo.ctlab.hict.hict_server.MainVerticle; + +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.Callable; +import java.util.function.Consumer; + +@Command( + name = "hict", + mixinStandardHelpOptions = true, + version = "HiCT JVM", + description = "HiCT server and conversion CLI.", + subcommands = { + HictCli.StartServer.class, + HictCli.StartApiServer.class, + HictCli.Convert.class + } +) +public class HictCli implements Runnable { + @Option(names = {"-v", "--verbose"}, description = "Enable verbose output.") + boolean verbose; + + @Override + public void run() { + CommandLine.usage(this, System.out); + } + + public static void main(final String[] args) { + final CommandLine commandLine = new CommandLine(new HictCli()) + .setCaseInsensitiveEnumValuesAllowed(true); + + if (args.length == 0) { + CommandLine.usage(commandLine.getCommand(), System.out); + commandLine.execute("start-server"); + return; + } + + final CommandLine.ParseResult parseResult = commandLine.parseArgs(args); + final var subcommand = parseResult.subcommand(); + final String subcommandName = subcommand != null ? subcommand.commandSpec().name() : ""; + + final int exitCode = commandLine.execute(args); + if ("start-server".equals(subcommandName) || "start-api-server".equals(subcommandName)) { + return; + } + System.exit(exitCode); + } + + @Command( + name = "start-server", + mixinStandardHelpOptions = true, + description = "Start API server with WebUI." + ) + static class StartServer implements Callable { + @Option(names = "--serve-webui", description = "Whether to serve WebUI (default: true).", defaultValue = "true") + boolean serveWebUi; + + @Override + public Integer call() { + System.setProperty("SERVE_WEBUI", Boolean.toString(serveWebUi)); + Launcher.main(new String[]{"run", MainVerticle.class.getName()}); + return 0; + } + } + + @Command( + name = "start-api-server", + mixinStandardHelpOptions = true, + description = "Start API server only (no WebUI)." + ) + static class StartApiServer implements Callable { + @Override + public Integer call() { + System.setProperty("SERVE_WEBUI", "false"); + Launcher.main(new String[]{"run", MainVerticle.class.getName()}); + return 0; + } + } + + @Command( + name = "convert", + mixinStandardHelpOptions = true, + description = "Run file converters.", + subcommands = { + HictCli.HictToMcool.class, + HictCli.McoolToHict.class + } + ) + static class Convert implements Runnable { + @Override + public void run() { + CommandLine.usage(this, System.out); + } + } + + abstract static class BaseConvert implements Callable { + @Option(names = {"-i", "--input"}, required = true, description = "Input file path.") + Path input; + + @Option(names = {"-o", "--output"}, required = true, description = "Output file path.") + Path output; + + @Option( + names = "--resolutions", + split = ",", + description = "Comma-separated list of resolutions to export (default: all in input)." + ) + List resolutions = new ArrayList<>(); + + @Option(names = "--chunk-size", defaultValue = "8192", description = "HDF5 chunk size (default: 8192).") + int chunkSize; + + @Option(names = "--compression", defaultValue = "0", description = "Compression level 0..9 (default: 0).") + int compression; + + @Option( + names = "--compression-algorithm", + defaultValue = "DEFLATE", + description = "Compression algorithm: ${COMPLETION-CANDIDATES} (default: ${DEFAULT-VALUE})." + ) + ConversionOptions.CompressionAlgorithm compressionAlgorithm; + + + @Option( + names = "--parallelism", + defaultValue = "0", + description = "Number of worker threads (0 or -1 = auto)." + ) + int parallelism; + + ConversionOptions toOptions(String agpPath, boolean applyAgp) { + return new ConversionOptions( + input, + output, + resolutions, + chunkSize, + compression, + compressionAlgorithm, + agpPath, + applyAgp, + parallelism + ); + } + + Consumer stdoutLogger() { + return message -> { + synchronized (System.out) { + System.out.println(message); + System.out.flush(); + } + }; + } + + void initializeHdf5() { + HDF5LibraryInitializer.initializeHDF5Library(); + } + } + + @Command( + name = "hict-to-mcool", + mixinStandardHelpOptions = true, + description = "Convert .hict.hdf5 to .mcool." + ) + static class HictToMcool extends BaseConvert { + @Option(names = "--agp", description = "AGP file path (optional).") + String agpPath = ConversionOptions.NO_AGP; + + @Option(names = "--apply-agp", description = "Apply AGP before export.") + boolean applyAgp; + + @Override + public Integer call() throws Exception { + initializeHdf5(); + new HictToMcoolConverter().convert(toOptions(agpPath, applyAgp), stdoutLogger()); + return 0; + } + } + + @Command( + name = "mcool-to-hict", + mixinStandardHelpOptions = true, + description = "Convert .mcool to .hict.hdf5." + ) + static class McoolToHict extends BaseConvert { + @Override + public Integer call() throws Exception { + initializeHdf5(); + new McoolToHictConverter().convert(toOptions(ConversionOptions.NO_AGP, false), stdoutLogger()); + return 0; + } + } +} diff --git a/src/main/java/ru/itmo/ctlab/hict/hict_server/util/shareable/ShareableWrappers.java b/src/main/java/ru/itmo/ctlab/hict/hict_server/util/shareable/ShareableWrappers.java index 3cbaf78..ded572c 100644 --- a/src/main/java/ru/itmo/ctlab/hict/hict_server/util/shareable/ShareableWrappers.java +++ b/src/main/java/ru/itmo/ctlab/hict/hict_server/util/shareable/ShareableWrappers.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/src/main/resources/libs/_sis-jhdf5-19.04.1.jar_ b/src/main/resources/libs/_sis-jhdf5-19.04.1.jar_ deleted file mode 100644 index 8937eee..0000000 Binary files a/src/main/resources/libs/_sis-jhdf5-19.04.1.jar_ and /dev/null differ diff --git a/src/test/java/ru/itmo/ctlab/hict/hict_server/TestMainVerticle.java b/src/test/java/ru/itmo/ctlab/hict/hict_server/TestMainVerticle.java index 993cf2f..ecad483 100644 --- a/src/test/java/ru/itmo/ctlab/hict/hict_server/TestMainVerticle.java +++ b/src/test/java/ru/itmo/ctlab/hict/hict_server/TestMainVerticle.java @@ -1,7 +1,7 @@ /* * MIT License * - * Copyright (c) 2021-2024. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. + * Copyright (c) 2021-2026. Aleksandr Serdiukov, Anton Zamyatin, Aleksandr Sinitsyn, Vitalii Dravgelis and Computer Technologies Laboratory ITMO University team. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal diff --git a/start.sh b/start.sh index 9071102..7f43f78 100755 --- a/start.sh +++ b/start.sh @@ -5,4 +5,4 @@ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:${HDF5_DIR}/lib:${HDF5_DIR}/lib/plugin" export HDF5_PLUGIN_PATH="${HDF5_DIR}/lib/plugin" export VERTXWEB_ENVIRONMENT="production" SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) -VXPORT=5000 DATA_DIR=. TILE_SIZE=256 java -jar "${SCRIPT_DIR}/hict_server-1.0.0-SNAPSHOT-fat.jar" +VXPORT=5000 DATA_DIR="${SCRIPT_DIR}/data/" TILE_SIZE=256 java -jar "${SCRIPT_DIR}/build/libs/hict_server-1.0.35-d1f2ade-webui_8060ecf-fat.jar" diff --git a/version.txt b/version.txt index 42c38b1..f193390 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.0.35-d1f2ade-webui_8060ecf +1.0.50-156c262-webui_d3840bd \ No newline at end of file