From efdfe31985fa3b36256e9845900ea1e1d96107d7 Mon Sep 17 00:00:00 2001 From: Esteban Zimanyi Date: Tue, 28 Jul 2026 22:00:05 +0200 Subject: [PATCH 1/2] Uniformize the JMEOS build with MobilitySpark and MobilityFlink MobilityKafka built the JMEOS jar through its own build-jmeos.sh and depended on it as com.mobilitydb:jmeos:1.4.0, while MobilitySpark and MobilityFlink build JMEOS main directly and install the jar as org.jmeos:meos:1.0. Adopt the same build and coordinates: CI checks out JMEOS, builds it against the provisioned catalog and libmeos, and installs jar/JMEOS.jar as org.jmeos:meos:1.0; the binding depends on those coordinates; the tests load libmeos from /usr/local/lib where provision-meos installs it. build-jmeos.sh is removed; GENERATION.md, README and the benchmark doc describe the shared build. --- .github/workflows/maven.yml | 34 ++++-- GENERATION.md | 53 +++++---- README.md | 16 +-- benchmark/docs/benchmark.md | 4 +- binding/pom.xml | 6 +- build-jmeos.sh | 212 ------------------------------------ pom.xml | 8 +- 7 files changed, 63 insertions(+), 270 deletions(-) delete mode 100755 build-jmeos.sh diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml index 18edb97..eaf9cbf 100644 --- a/.github/workflows/maven.yml +++ b/.github/workflows/maven.yml @@ -26,7 +26,7 @@ jobs: # provision-meos action. Tracking master (not a pinned commit) keeps the # source, the catalog, the JMEOS jar and the libmeos the tests load all moving # together, so the generated facades can never drift from the surface they run - # against. build-jmeos.sh then only builds the JMEOS jar against this libmeos. + # against. The JMEOS jar is then built from JMEOS main against this libmeos. - name: Provision MEOS catalog + libmeos id: provision uses: MobilityDB/MEOS-API/.github/actions/provision-meos@master @@ -39,14 +39,28 @@ jobs: # The catalog is gitignored — it only ever exists as a build artifact. run: cp "${{ steps.provision.outputs.catalog-path }}" tools/meos-idl.json - - name: Build JMEOS.jar against the installed libmeos - # libmeos is already built and installed under /usr/local by the - # provision-meos action, so build-jmeos.sh skips its own libmeos build - # and only builds the JMEOS jar from JMEOS main, installs it into the - # local Maven repo, and stages libmeos.so under the reactor-root lib/. - run: JOBS=2 ./build-jmeos.sh + - name: Check out JMEOS + uses: actions/checkout@v4 + with: + repository: MobilityDB/JMEOS + path: jmeos + + - name: Build + install the JMEOS jar as org.jmeos:meos:1.0 + # Mirror MobilitySpark and MobilityFlink: stage the derived catalog + libmeos into the + # JMEOS checkout and build it (jmeos-core's generate-sources emits GeneratedFunctions + # from the catalog), then install the plain jar/JMEOS.jar under the org.jmeos:meos:1.0 + # coordinates the binding depends on. Tests are skipped — JMEOS's own CI exercises its + # FFI suite; only the jar is needed here. + run: | + mkdir -p jmeos/codegen/input + cp "${{ steps.provision.outputs.catalog-path }}" jmeos/codegen/input/meos-idl.json + cp /usr/local/lib/libmeos.so jmeos/jmeos-core/src/ + mvn -B -f jmeos/pom.xml clean install -Dmaven.test.skip=true + mvn -B install:install-file \ + -Dfile=jmeos/jar/JMEOS.jar \ + -DgroupId=org.jmeos -DartifactId=meos -Dversion=1.0 -Dpackaging=jar - name: Test the Kafka Streams app - # The generated MeosOps* facades forward to the installed JMEOS jar; the tests - # exercise them against the freshly built libmeos, resolved via LD_LIBRARY_PATH. - run: LD_LIBRARY_PATH="$PWD/lib" mvn -B test + # The generated MeosOps* facades forward to the installed JMEOS jar; the tests exercise + # them against libmeos from /usr/local/lib (surefire sets LD_LIBRARY_PATH=meos.lib.dir). + run: mvn -B clean test diff --git a/GENERATION.md b/GENERATION.md index ac4b48d..dc37978 100644 --- a/GENERATION.md +++ b/GENERATION.md @@ -43,31 +43,30 @@ and `build-helper` adds it as a source root. The sole hand-written class under ## The build chain — no committed binaries -`build-jmeos.sh` reproduces the native/JVM chain, so the repository carries no jar or -`libmeos.so`: +The chain is reproduced from source, so the repository carries no jar or `libmeos.so`: ``` MobilityDB @ master → provision-meos (MEOS-API/run.py + cmake -DMEOS=ON -DALL) → meos-idl.json + libmeos.so - → build-jmeos.sh: JMEOS main → JMEOS.jar → com.mobilitydb:jmeos:1.4.0 + → JMEOS main (mvn install) → JMEOS.jar → org.jmeos:meos:1.0 → tools/codegen_jvm.py --engine kafka (full jar surface) → org.mobilitydb.meos.MeosOps* facades - → kafka-streams-app (mvn test) + → binding (mvn test) ``` -CI derives the catalog + all-families `libmeos.so` from upstream MobilityDB master through -the shared `provision-meos` action, stages the catalog to `tools/meos-idl.json`, and lets -`build-jmeos.sh` build the JMEOS jar from JMEOS `main` against that libmeos. Tracking master -(not a pinned commit) keeps the source, the catalog, the jar and the libmeos the tests load -all moving together, so the generated facades can never drift from the surface they run -against. +CI derives the catalog + all-families `libmeos.so` from upstream MobilityDB master through the +shared `provision-meos` action, stages the catalog to `tools/meos-idl.json`, and builds the +JMEOS jar from JMEOS `main` against that libmeos, installing it as `org.jmeos:meos:1.0` — the +same jar coordinates and build steps MobilitySpark and MobilityFlink use. Tracking master (not a +pinned commit) keeps the source, the catalog, the jar and the libmeos the tests load all moving +together, so the generated facades can never drift from the surface they run against. ## Regenerating by hand -CI performs the steps below via `provision-meos` and `build-jmeos.sh`. To run them yourself -you need a JDK, Maven, CMake and the MEOS build dependencies. +CI performs the steps below via `provision-meos`. To run them yourself you need a JDK, Maven, +CMake and the MEOS build dependencies. **1. Derive libmeos and the catalog from MobilityDB master.** Both come from one commit; see -`MEOS-API/GENERATION.md` for the two commands: +`MEOS-API/GENERATION.md` for the commands: ```bash MDB=~/src/MobilityDB # checkout at the commit you are deriving from @@ -78,28 +77,26 @@ cmake --install "$MDB/build" --prefix "$MDB/.prefix" cd "$MEOSAPI" && MDB_SRC_ROOT="$MDB" python3 run.py "$MDB/.prefix/include" ``` -**2. Stage the catalog and build the JMEOS jar.** `tools/meos-idl.json` is derived, not -committed. `build-jmeos.sh` builds the jar from JMEOS `main` and installs it into the local -Maven repository as `com.mobilitydb:jmeos:1.4.0`, the coordinates this build resolves. It -reuses an already-built libmeos rather than building its own — `INSTALLED_LIBMEOS` points at -one, defaulting to `/usr/local/lib/libmeos.so` — and copies it into `kafka-streams-app/lib/`: +**2. Build the JMEOS jar against that catalog and install it into the local Maven repository** +under the coordinates this build resolves — `org.jmeos:meos:1.0`, the same as MobilitySpark and +MobilityFlink: ```bash -cd ~/src/MobilityKafka -cp "$MEOSAPI/output/meos-idl.json" tools/meos-idl.json -INSTALLED_LIBMEOS="$MDB/.prefix/lib/libmeos.so" ./build-jmeos.sh +cd ~/src/JMEOS # JMEOS main +CATALOG="$MEOSAPI/output/meos-idl.json" LIBMEOS="$MDB/.prefix/lib/libmeos.so" \ + tools/regen-from-catalog.sh +mvn install:install-file -Dfile=jar/JMEOS.jar \ + -DgroupId=org.jmeos -DartifactId=meos -Dversion=1.0 -Dpackaging=jar ``` -Without `INSTALLED_LIBMEOS` the script clones MobilityDB at `MOBILITYDB_REF` (default -`master`) and builds libmeos itself, which is the same derivation done twice. - -**3. Build and run the tests**, resolving the library from where step 2 placed it: +**3. Stage the catalog and build.** `tools/meos-idl.json` is derived, not committed: ```bash -cd kafka-streams-app -LD_LIBRARY_PATH="$PWD/lib" mvn test +cd ~/src/MobilityKafka +cp "$MEOSAPI/output/meos-idl.json" tools/meos-idl.json +mvn -Dmeos.lib.dir="$MDB/.prefix/lib" clean test ``` `generate-sources` runs `tools/codegen_jvm.py --engine kafka --catalog ../tools/meos-idl.json --jar --out target/generated-facades`, so the `MeosOps*` facades are -regenerated by the build itself. `LD_LIBRARY_PATH` is how the tests find `libmeos.so`. +regenerated by the build itself. `meos.lib.dir` is where the tests find `libmeos.so`. diff --git a/README.md b/README.md index 9238699..c39a206 100644 --- a/README.md +++ b/README.md @@ -53,17 +53,11 @@ Spatial predicates today use pure-Java great-circle (`Haversine`) and planar seg The spatial predicates route through MEOS via the [JMEOS](https://github.com/MobilityDB/JMEOS) bridge, so the build needs the JMEOS jar and the native `libmeos.so`. Neither is -committed to this repository — generate them from source with the helper script, -which clones MobilityDB and JMEOS at pinned, immutable refs, builds `libmeos.so`, -builds the jar, installs the jar into the local Maven repository, and stages -`libmeos.so` for the runtime: - -``` -./build-jmeos.sh -``` - -Run it once (re-run it only to bump the pinned MobilityDB/JMEOS refs at the top of -the script). After it succeeds, JMEOS resolves as an ordinary Maven dependency. +committed to this repository — both are derived from source: derive `libmeos.so` +and the catalog from MobilityDB master, build the JMEOS jar and install it as +`org.jmeos:meos:1.0`, then build. `GENERATION.md` gives the exact commands, the same +ones MobilitySpark and MobilityFlink use. Once the jar is installed, JMEOS resolves +as an ordinary Maven dependency. ### Build the app diff --git a/benchmark/docs/benchmark.md b/benchmark/docs/benchmark.md index 5e92692..9ad3778 100644 --- a/benchmark/docs/benchmark.md +++ b/benchmark/docs/benchmark.md @@ -38,8 +38,8 @@ gone idle. The trailing settle time is excluded from the wall; each consumed record runs the cell's MEOS predicate, so this is the steady-state per-event processing rate, directly comparable to the MobilityFlink figures. -Run from `kafka-streams-app/` after `../build-jmeos.sh` (which installs JMEOS into -the local Maven repository and stages `libmeos.so` under `lib/`). The test-scope +Run after building the chain (see `GENERATION.md`), which installs JMEOS as +`org.jmeos:meos:1.0` and libmeos under `/usr/local/lib`. The test-scope classpath that `mvn` reconstructs already carries JMEOS and the embedded broker, so no jar is referenced by path: diff --git a/binding/pom.xml b/binding/pom.xml index 915db5d..ab66233 100644 --- a/binding/pom.xml +++ b/binding/pom.xml @@ -18,8 +18,8 @@ - com.mobilitydb - jmeos + org.jmeos + meos ${jmeos.version} @@ -87,7 +87,7 @@ --catalog ${project.basedir}/../tools/meos-idl.json --jar - ${settings.localRepository}/com/mobilitydb/jmeos/${jmeos.version}/jmeos-${jmeos.version}.jar + ${settings.localRepository}/org/jmeos/meos/${jmeos.version}/meos-${jmeos.version}.jar --out ${project.build.directory}/generated-facades diff --git a/build-jmeos.sh b/build-jmeos.sh deleted file mode 100755 index 6c7cd9e..0000000 --- a/build-jmeos.sh +++ /dev/null @@ -1,212 +0,0 @@ -#!/usr/bin/env bash -# -# build-jmeos.sh — build the JMEOS jar and the native libmeos.so from source and -# install them locally, so the repository never has to carry the binaries. -# -# This is the downstream generation chain for the JVM streaming tools: -# -# MobilityDB/MEOS (deliverable PRs) -> JMEOS (FFI facade + jar) -> MobilityKafka -# -# (The MEOS-API meos-idl.json step is pre-materialized in the JMEOS branch's -# committed codegen/input/meos-idl.json, so this script only has to build the -# two endpoints.) -# -# What it does: -# 1. Clones MobilityDB at the pinned ref and builds libmeos.so (cmake -DMEOS=ON). -# 2. Clones JMEOS at the pinned ref, drops libmeos.so in, and builds JMEOS.jar. -# 3. Registers the jar in the local Maven repository via -# `mvn install:install-file` under the coordinates the kafka-streams-app -# pom depends on (com.mobilitydb:jmeos:1.4.0 by default). -# 4. Copies libmeos.so into the reactor-root lib/ for the test/runtime -# LD_LIBRARY_PATH. -# -# After running this once, `mvn test` resolves JMEOS as -# an ordinary dependency — no committed jar/so required. -# -# The refs below track upstream MobilityDB master and MobilityDB/JMEOS main — the -# surfaces this project is generated against — as branch names (overridable env -# vars), so a build always picks up the current master/main tips. -# -set -euo pipefail - -# --------------------------------------------------------------------------- -# Pinned sources (override any of these via the environment). -# --------------------------------------------------------------------------- -# MobilityDB master — the surface the JMEOS facade is generated against, so the -# built libmeos.so matches the facade catalog. Overridable via the environment. -MOBILITYDB_REPO="${MOBILITYDB_REPO:-https://github.com/MobilityDB/MobilityDB.git}" -MOBILITYDB_REF="${MOBILITYDB_REF:-master}" # track upstream master (only used when libmeos is not pre-installed) - -# JMEOS main — functions.GeneratedFunctions (built at build-time from the committed -# catalog) plus the org.mobilitydb.meos.MeosOps* facades. -JMEOS_REPO="${JMEOS_REPO:-https://github.com/MobilityDB/JMEOS.git}" -JMEOS_REF="${JMEOS_REF:-main}" # track upstream main - -# Maven coordinates the jar is installed under (must match kafka-streams-app/pom.xml). -JMEOS_GROUP_ID="${JMEOS_GROUP_ID:-com.mobilitydb}" -JMEOS_ARTIFACT_ID="${JMEOS_ARTIFACT_ID:-jmeos}" -JMEOS_VERSION="${JMEOS_VERSION:-1.4.0}" - -# --------------------------------------------------------------------------- -# Layout. -# --------------------------------------------------------------------------- -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -APP_DIR="${SCRIPT_DIR}" -WORK_DIR="${WORK_DIR:-${SCRIPT_DIR}/.build-jmeos}" -JOBS="${JOBS:-$(nproc 2>/dev/null || echo 4)}" - -log() { printf '\n\033[1;34m==>\033[0m %s\n' "$*"; } - -# --------------------------------------------------------------------------- -# Preconditions. -# --------------------------------------------------------------------------- -for tool in git cmake make mvn; do - command -v "$tool" >/dev/null 2>&1 || { echo "error: '$tool' is required but not on PATH" >&2; exit 1; } -done - -mkdir -p "${WORK_DIR}" - -# clone_at -# Clones (or reuses) and checks out the exact . may be a tag, -# branch or commit SHA; PR-head SHAs are fetched from the pull ref namespace if -# they are not reachable from the default branches. -clone_at() { - local repo="$1" ref="$2" dest="$3" - if [ ! -d "${dest}/.git" ]; then - log "Cloning ${repo}" - git clone "${repo}" "${dest}" - fi - # Honor a changed repo URL on a reused checkout (e.g. fork -> upstream). - git -C "${dest}" remote set-url origin "${repo}" - git -C "${dest}" fetch --quiet --tags origin - if ! git -C "${dest}" cat-file -e "${ref}^{commit}" 2>/dev/null; then - # Not reachable from the default branches — fetch the exact ref directly, then - # fall back to the PR-head namespace for an open-PR SHA. - git -C "${dest}" fetch --quiet origin "${ref}" 2>/dev/null || \ - git -C "${dest}" fetch --quiet origin '+refs/pull/*/head:refs/remotes/origin/pr/*' || true - fi - log "Checking out ${ref}" - # -f: a prior run copies libmeos.so into jmeos-core/src/, dirtying the tree; discard - # such local changes so a reused checkout does not block on them. - git -C "${dest}" -c advice.detachedHead=false checkout -f --quiet "${ref}" -} - -# --------------------------------------------------------------------------- -# 1. Obtain libmeos.so. -# --------------------------------------------------------------------------- -# When a libmeos is already installed — e.g. CI builds and installs it through -# the shared MobilityDB/MEOS-API provision-meos action, whose `cmake --install` -# also provisions spatial_ref_sys.csv / ways1000.csv under /usr/local/share — -# reuse it and build only the JMEOS jar against it. Otherwise build libmeos from -# the pinned MobilityDB commit here (and stage the reference data explicitly). -INSTALLED_LIBMEOS="${INSTALLED_LIBMEOS:-/usr/local/lib/libmeos.so}" -if [ -f "${INSTALLED_LIBMEOS}" ]; then - LIBMEOS_SO="${INSTALLED_LIBMEOS}" - log "Reusing pre-installed libmeos: ${LIBMEOS_SO} (skipping the MobilityDB build)" -else - MDB_DIR="${WORK_DIR}/MobilityDB" - clone_at "${MOBILITYDB_REPO}" "${MOBILITYDB_REF}" "${MDB_DIR}" - - # Build every optional MEOS family via -DALL so the facades link against a - # libmeos with the full symbol surface (circular buffers, H3, JSON, network - # points, pgPointCloud, geoposes, quadbin, raster, rigid geometries, Arrow). - # H3 is pinned to the distro's system library (the CI apt step installs - # libh3-dev); pgPointCloud's vendored libpc.a needs pg_config, pinned to the - # apt.postgresql.org PostgreSQL 17 the CI workflow installs. Override via - # MEOS_CMAKE_ARGS for a non-Debian layout. - MEOS_CMAKE_ARGS="${MEOS_CMAKE_ARGS:--DALL=ON -DH3_LIBRARY=/usr/lib/x86_64-linux-gnu/libh3.so -DH3_INCLUDE_DIR=/usr/include/h3 -DPOSTGRESQL_PG_CONFIG=/usr/lib/postgresql/17/bin/pg_config}" - - log "Building libmeos.so (MEOS=ON ${MEOS_CMAKE_ARGS})" - rm -rf "${MDB_DIR}/build" - cmake -S "${MDB_DIR}" -B "${MDB_DIR}/build" -DMEOS=ON ${MEOS_CMAKE_ARGS} >/dev/null - cmake --build "${MDB_DIR}/build" --target meos -j "${JOBS}" - - LIBMEOS_SO="$(find "${MDB_DIR}/build" -name 'libmeos.so' -print -quit)" - [ -n "${LIBMEOS_SO}" ] || { echo "error: libmeos.so not produced by the MEOS build" >&2; exit 1; } - log "Built ${LIBMEOS_SO}" - - # ------------------------------------------------------------------------- - # 1b. Provision MEOS's SRID/network reference data at its default runtime path. - # ------------------------------------------------------------------------- - # libmeos resolves SRIDs by reading spatial_ref_sys.csv from a fixed default - # path (meos/src/geo/tspatial_transform_meos.c: SPATIAL_REF_SYS_CSV = - # "/usr/local/share/spatial_ref_sys.csv"). A full `cmake --install` would place - # it there (meos/CMakeLists.txt), but this branch only builds the `meos` target, - # so the data files are staged explicitly. Without them, any SRID-touching call - # (npoint, tgeompoint) makes libmeos print "Cannot open the spatial_ref_sys.csv - # file" to stdout, which corrupts the surefire fork channel and terminates the - # JVM under test ("The forked VM terminated without properly saying goodbye"). - MEOS_DATA_DIR="${MEOS_DATA_DIR:-/usr/local/share}" - log "Provisioning MEOS reference data into ${MEOS_DATA_DIR}" - provision_data() { - local src="$1" dst="$2" - install -Dm644 "${src}" "${dst}" 2>/dev/null || sudo install -Dm644 "${src}" "${dst}" - } - provision_data "${MDB_DIR}/meos/src/geo/spatial_ref_sys.csv" "${MEOS_DATA_DIR}/spatial_ref_sys.csv" - provision_data "${MDB_DIR}/meos/examples/data/ways1000.csv" "${MEOS_DATA_DIR}/ways1000.csv" -fi - -# --------------------------------------------------------------------------- -# 2. Build JMEOS.jar against that libmeos.so. -# --------------------------------------------------------------------------- -JMEOS_DIR="${WORK_DIR}/JMEOS" -clone_at "${JMEOS_REPO}" "${JMEOS_REF}" "${JMEOS_DIR}" - -# JMEOS' build bundles src/libmeos.so into the jar and JarLibraryLoader extracts it. -cp -f "${LIBMEOS_SO}" "${JMEOS_DIR}/jmeos-core/src/libmeos.so" - -# JMEOS main derives functions.GeneratedFunctions from codegen/input/meos-idl.json -# but tracks the MEOS-API catalog (it no longer commits it). Stage the catalog -# derived by provision-meos — the workflow puts it at tools/meos-idl.json — so the -# jar is built from the same master surface as the libmeos and the facades. -MEOS_CATALOG="${MEOS_CATALOG:-${SCRIPT_DIR}/tools/meos-idl.json}" -if [ -s "${MEOS_CATALOG}" ]; then - mkdir -p "${JMEOS_DIR}/codegen/input" - cp -f "${MEOS_CATALOG}" "${JMEOS_DIR}/codegen/input/meos-idl.json" - log "Staged MEOS catalog into ${JMEOS_DIR}/codegen/input/meos-idl.json" -else - echo "error: MEOS catalog not found at ${MEOS_CATALOG}; stage it (provision-meos) first" >&2 - exit 1 -fi - -log "Building JMEOS.jar" -# FunctionsGenerator lives in the codegen module, which jmeos-core does not -# declare as a Maven dependency — so '-am' will not build it. Compile it first -# so jmeos-core's build-time facade generation can find it. Use -# 'maven.test.skip' (not 'skipTests'): the jmeos-core pom hardcodes -# false, which overrides -DskipTests but not this. -mvn -f "${JMEOS_DIR}/pom.xml" -q -pl codegen compile -mvn -f "${JMEOS_DIR}/pom.xml" -q -pl jmeos-core -am -Dmaven.test.skip=true package - -JMEOS_JAR="${JMEOS_DIR}/jar/JMEOS.jar" -[ -f "${JMEOS_JAR}" ] || { echo "error: ${JMEOS_JAR} was not produced" >&2; exit 1; } - -# --------------------------------------------------------------------------- -# 3. Install the jar into the local Maven repository. -# --------------------------------------------------------------------------- -log "Installing ${JMEOS_GROUP_ID}:${JMEOS_ARTIFACT_ID}:${JMEOS_VERSION} into the local Maven repo" -mvn -q install:install-file \ - -Dfile="${JMEOS_JAR}" \ - -DgroupId="${JMEOS_GROUP_ID}" \ - -DartifactId="${JMEOS_ARTIFACT_ID}" \ - -Dversion="${JMEOS_VERSION}" \ - -Dpackaging=jar - -# --------------------------------------------------------------------------- -# 4. Stage libmeos.so for the kafka-streams-app runtime (LD_LIBRARY_PATH). -# --------------------------------------------------------------------------- -mkdir -p "${APP_DIR}/lib" -cp -f "${LIBMEOS_SO}" "${APP_DIR}/lib/libmeos.so" - -log "Done." -cat <2.18.2 2.0.16 5.11.4 - 1.4.0 + 1.0 2.1.10 true - - ${maven.multiModuleProjectDirectory}/lib + + /usr/local/lib From 594305e428e5cc3a2a41a2a7f51b8981719e190c Mon Sep 17 00:00:00 2001 From: Esteban Zimanyi Date: Tue, 28 Jul 2026 22:03:31 +0200 Subject: [PATCH 2/2] Add a one-command refresh against upstream MEOS tools/refresh-from-master.sh refreshes the MeosOps* facades against upstream MEOS in one command: it runs the shared MEOS-API refresh-jvm-chain.sh over this repo, deriving libmeos and the catalog from MobilityDB master, building the JMEOS jar, and regenerating the facades. tools/refresh.conf holds this consumer last leg (the repo-root module, the org.jmeos:meos:1.0 coordinates, the mvn command). GENERATION.md leads with the one-liner; .gitignore excludes the .meos-chain scratch. --- .gitignore | 3 +++ GENERATION.md | 45 ++++++++++++------------------------ tools/refresh-from-master.sh | 35 ++++++++++++++++++++++++++++ tools/refresh.conf | 8 +++++++ 4 files changed, 61 insertions(+), 30 deletions(-) create mode 100755 tools/refresh-from-master.sh create mode 100644 tools/refresh.conf diff --git a/.gitignore b/.gitignore index d88fedd..05b925b 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,9 @@ target/ # MobilityDB master (provision-meos), never committed. tools/meos-idl.json +# tools/refresh-from-master.sh scratch (sibling clones + libmeos prefix). +.meos-chain/ + # JMEOS native dependency — generated by build-jmeos.sh, never committed .build-jmeos/ jar/ diff --git a/GENERATION.md b/GENERATION.md index dc37978..bc6e655 100644 --- a/GENERATION.md +++ b/GENERATION.md @@ -60,43 +60,28 @@ same jar coordinates and build steps MobilitySpark and MobilityFlink use. Tracki pinned commit) keeps the source, the catalog, the jar and the libmeos the tests load all moving together, so the generated facades can never drift from the surface they run against. -## Regenerating by hand +## Refreshing against upstream MEOS -CI performs the steps below via `provision-meos`. To run them yourself you need a JDK, Maven, -CMake and the MEOS build dependencies. - -**1. Derive libmeos and the catalog from MobilityDB master.** Both come from one commit; see -`MEOS-API/GENERATION.md` for the commands: +`tools/refresh-from-master.sh` runs the whole chain — deriving libmeos and the catalog from +MobilityDB master, building the JMEOS jar, and regenerating the `MeosOps*` facades — in one +command, on Ubuntu with a JDK, Maven, CMake and the MEOS build dependencies installed: ```bash -MDB=~/src/MobilityDB # checkout at the commit you are deriving from -MEOSAPI=~/src/MEOS-API -cmake -S "$MDB" -B "$MDB/build" -DCMAKE_BUILD_TYPE=Release -DMEOS=ON -DALL=ON -cmake --build "$MDB/build" -j"$(nproc)" -cmake --install "$MDB/build" --prefix "$MDB/.prefix" -cd "$MEOSAPI" && MDB_SRC_ROOT="$MDB" python3 run.py "$MDB/.prefix/include" +tools/refresh-from-master.sh ``` -**2. Build the JMEOS jar against that catalog and install it into the local Maven repository** -under the coordinates this build resolves — `org.jmeos:meos:1.0`, the same as MobilitySpark and -MobilityFlink: +It runs the shared `MobilityDB/MEOS-API` `refresh-jvm-chain.sh` over this repo (cloning MEOS-API +master the first time). All of that script's options pass through: ```bash -cd ~/src/JMEOS # JMEOS main -CATALOG="$MEOSAPI/output/meos-idl.json" LIBMEOS="$MDB/.prefix/lib/libmeos.so" \ - tools/regen-from-catalog.sh -mvn install:install-file -Dfile=jar/JMEOS.jar \ - -DgroupId=org.jmeos -DartifactId=meos -Dversion=1.0 -Dpackaging=jar +tools/refresh-from-master.sh --mdb ~/src/MobilityDB # refresh against a local MobilityDB branch +tools/refresh-from-master.sh --skip-tests # regenerate + compile, skip the tests ``` -**3. Stage the catalog and build.** `tools/meos-idl.json` is derived, not committed: - -```bash -cd ~/src/MobilityKafka -cp "$MEOSAPI/output/meos-idl.json" tools/meos-idl.json -mvn -Dmeos.lib.dir="$MDB/.prefix/lib" clean test -``` +This repo's last leg is `tools/refresh.conf`: the repo-root Maven module, the `org.jmeos:meos:1.0` +jar coordinates, and the `mvn … clean test` command. `generate-sources` runs `tools/codegen_jvm.py +--engine kafka` over the staged `tools/meos-idl.json` and the installed jar, so the `MeosOps*` +facades are regenerated by the build itself; `meos.lib.dir` is where the tests find `libmeos.so`. -`generate-sources` runs `tools/codegen_jvm.py --engine kafka --catalog ../tools/meos-idl.json ---jar --out target/generated-facades`, so the `MeosOps*` facades are -regenerated by the build itself. `meos.lib.dir` is where the tests find `libmeos.so`. +The chain's per-leg commands, which `refresh-jvm-chain.sh` composes, are in `MEOS-API/GENERATION.md` +(`provision-meos.sh`) and `JMEOS/GENERATION.md` (`regen-from-catalog.sh`). diff --git a/tools/refresh-from-master.sh b/tools/refresh-from-master.sh new file mode 100755 index 0000000..26e9155 --- /dev/null +++ b/tools/refresh-from-master.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# refresh-from-master.sh — refresh this consumer's MEOS facades against the latest MEOS API, +# end to end, with one command: +# +# tools/refresh-from-master.sh +# +# It runs the shared MobilityDB/MEOS-API refresh-jvm-chain.sh over this repo — deriving the +# catalog and libmeos from the latest MobilityDB master, building the JMEOS jar, and +# regenerating this consumer's facades. The per-consumer last leg is in tools/refresh.conf. +# +# All of refresh-jvm-chain.sh's options pass through, e.g.: +# tools/refresh-from-master.sh --mdb ~/src/MobilityDB # refresh against a local MobilityDB branch +# tools/refresh-from-master.sh --skip-tests # regenerate + compile, skip the test run +# +# MEOSAPI= uses an existing MEOS-API checkout (any branch); otherwise MEOS-API master is +# cloned into the work dir. WORK_DIR overrides the scratch location (default /.meos-chain). +set -euo pipefail + +HERE="$(cd "$(dirname "$0")/.." && pwd)" +WORK="${WORK_DIR:-$HERE/.meos-chain}" +MEOSAPI="${MEOSAPI:-}" + +if [ -z "$MEOSAPI" ]; then + MEOSAPI="$WORK/MEOS-API" + mkdir -p "$WORK" + if [ -d "$MEOSAPI/.git" ]; then + git -C "$MEOSAPI" fetch --quiet https://github.com/MobilityDB/MEOS-API master + git -C "$MEOSAPI" checkout --quiet FETCH_HEAD + else + git clone --quiet https://github.com/MobilityDB/MEOS-API "$MEOSAPI" + fi +fi + +exec "$MEOSAPI/tools/refresh-jvm-chain.sh" \ + --consumer "$HERE" --meos-api "$MEOSAPI" --work-dir "$WORK" "$@" diff --git a/tools/refresh.conf b/tools/refresh.conf new file mode 100644 index 0000000..68e8a2a --- /dev/null +++ b/tools/refresh.conf @@ -0,0 +1,8 @@ +# refresh.conf — MobilityKafka's last leg for tools/refresh-jvm-chain.sh (in MEOS-API). +# The chain (MobilityDB -> catalog + libmeos -> JMEOS jar) is shared; only these differ per +# consumer. BUILD_CMD runs from / with $PREFIX (the libmeos install prefix), +# $CATALOG and $JAR exported; $SKIP_TESTS is set when --skip-tests is passed. +ENGINE=kafka +BUILD_DIR=. +JMEOS_COORDS=org.jmeos:meos:1.0 +BUILD_CMD='mvn -B -Dmeos.lib.dir="$PREFIX/lib" ${SKIP_TESTS:+-DskipTests} clean test'