diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index e0c005ef..6bc6171a 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -79,4 +79,30 @@ Common types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`. - **Docker**: Many components rely on Docker for container orchestration during development (e.g., running MongoDB or Analysis instances). +## Release Strategy + +ScriptBee uses a namespaced tagging strategy to manage independent component releases within the monorepo. + +### Tag Naming Convention + +| Tag Format | Component | Artifact(s) | GitHub Release | +|:-----------------------|:-----------------------------------|:--------------------------------------|:---------------| +| `v` | Unified Application (Backend + UI) | Docker: `dxworks/scriptbee` | **Yes (Main)** | +| `analysis@` | Analysis Microservice | Docker: `dxworks/scriptbee/analysis` | No (Silent) | +| `plugin-api@` | Plugin API | NuGet: `DxWorks.ScriptBee.Plugin.Api` | **Yes** | +| `bundle@` | Default Plugin Bundle | Zip Archive | **Yes** | + +### Release Process + +1. **Tagging**: Push a tag matching one of the patterns above (e.g., `git tag v1.2.3 && git push origin v1.2.3`). +2. **Automation**: GitHub Actions will automatically: + - Run the relevant test suites. + - Build and publish the artifacts to Docker Hub or NuGet. + - Draft a GitHub Release with auto-generated release notes (for `v*`, `plugin-api@`, and `bundle@`). +3. **Manual Finalization**: The maintainer should review the drafted GitHub Release, refine the notes if necessary, and + publish it. + +> All releases are intended to be backward compatible. If a breaking change is required, it must be clearly documented +> in the release notes. + Thank you for your contributions! diff --git a/.github/workflows/release-analysis.yml b/.github/workflows/release-analysis.yml new file mode 100644 index 00000000..94f2a944 --- /dev/null +++ b/.github/workflows/release-analysis.yml @@ -0,0 +1,68 @@ +name: Release Analysis + +on: + push: + tags: + - 'analysis@*' + +jobs: + backend-test: + name: Backend Test + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v6 + + - name: Setup .NET + uses: actions/setup-dotnet@v4 + with: + dotnet-version: '10.0.x' + + - name: Restore dependencies + run: dotnet restore ScriptBee.slnx + + - name: Build + run: dotnet build ScriptBee.slnx --no-restore + + - name: Test + run: dotnet test ScriptBee.slnx --no-build --verbosity normal + + analysis-build-push: + name: Analysis Build & Push Docker + needs: backend-test + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + + steps: + - name: Checkout code + uses: actions/checkout@v6 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKER_USERNAME }} + password: ${{ secrets.DOCKER_PASSWORD }} + + - name: Extract version from tag + id: meta + run: | + VERSION="${GITHUB_REF_NAME#*@}" + echo "version=$VERSION" >> $GITHUB_OUTPUT + + - name: Build and push Analysis Docker image + uses: docker/build-push-action@v7 + with: + context: . + file: ./analysis.Dockerfile + push: true + tags: | + dxworks/scriptbee/analysis:${{ steps.meta.outputs.version }} + dxworks/scriptbee/analysis:latest + cache-from: type=gha + cache-to: type=gha,mode=max diff --git a/.github/workflows/release-default-bundle.yml b/.github/workflows/release-default-bundle.yml index 1a128a0c..2ce58ea2 100644 --- a/.github/workflows/release-default-bundle.yml +++ b/.github/workflows/release-default-bundle.yml @@ -3,7 +3,7 @@ name: Release Default Bundle on: push: tags: - - 'bundle/v*' + - 'bundle@*' jobs: build-and-publish: @@ -22,9 +22,9 @@ jobs: - name: Extract version from tag id: meta run: | - TAG="${GITHUB_REF#refs/tags/bundle/v}" - echo "version=$TAG" >> $GITHUB_OUTPUT - echo "zip_name=scriptbee-default-bundle-$TAG.zip" >> $GITHUB_OUTPUT + VERSION="${GITHUB_REF_NAME#*@}" + echo "version=$VERSION" >> $GITHUB_OUTPUT + echo "zip_name=scriptbee-default-bundle-$VERSION.zip" >> $GITHUB_OUTPUT - name: Install zip run: sudo apt-get install -y zip @@ -50,9 +50,9 @@ jobs: - name: Extract version from tag id: meta run: | - TAG="${GITHUB_REF#refs/tags/bundle/v}" - echo "version=$TAG" >> $GITHUB_OUTPUT - echo "zip_name=scriptbee-default-bundle-$TAG.zip" >> $GITHUB_OUTPUT + VERSION="${GITHUB_REF_NAME#*@}" + echo "version=$VERSION" >> $GITHUB_OUTPUT + echo "zip_name=scriptbee-default-bundle-$VERSION.zip" >> $GITHUB_OUTPUT - name: Download bundle artifact uses: actions/download-artifact@v4 diff --git a/.github/workflows/release-plugin-api.yml b/.github/workflows/release-plugin-api.yml index 474084ee..1c3e34b5 100644 --- a/.github/workflows/release-plugin-api.yml +++ b/.github/workflows/release-plugin-api.yml @@ -3,7 +3,7 @@ name: Release Plugin API on: push: tags: - - 'plugin-api/v*' + - 'plugin-api@*' jobs: build-and-publish: @@ -22,8 +22,8 @@ jobs: - name: Extract version from tag id: meta run: | - TAG="${GITHUB_REF#refs/tags/plugin-api/v}" - echo "version=$TAG" >> $GITHUB_OUTPUT + VERSION="${GITHUB_REF_NAME#*@}" + echo "version=$VERSION" >> $GITHUB_OUTPUT - name: Restore dependencies run: dotnet restore DxWorks.ScriptBee.Plugin.Api/DxWorks.ScriptBee.Plugin.Api.csproj diff --git a/.github/workflows/release.yml b/.github/workflows/release-server.yml similarity index 56% rename from .github/workflows/release.yml rename to .github/workflows/release-server.yml index 070b6fc3..ea257a79 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release-server.yml @@ -1,9 +1,9 @@ -name: Release +name: Release Server on: push: tags: - - 'v*' + - 'v[0-9]*' jobs: # ------------------------------------------------------------------ @@ -35,7 +35,7 @@ jobs: shell: bash run: | PLAYWRIGHT_VERSION=$(npm list @vitest/browser-playwright --json | jq -r '.dependencies["@vitest/browser-playwright"].version') - echo "PLAYWRIGHT_VERSION=$PLAYWRIGHT_VERSION" >> $GITHUB_ENV + echo "version=$PLAYWRIGHT_VERSION" >> $GITHUB_OUTPUT - name: Cache Playwright binaries uses: actions/cache@v5 @@ -43,7 +43,7 @@ jobs: with: path: | ~/.cache/ms-playwright - key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }} + key: ${{ runner.os }}-playwright-${{ steps.playwright-version.outputs.version }} - name: Install Playwright Browsers if: steps.playwright-cache.outputs.cache-hit != 'true' @@ -55,42 +55,6 @@ jobs: - name: Run tests run: npm run test:ci - ui-build-push: - name: UI Build & Push Docker - needs: ui-test - runs-on: ubuntu-latest - permissions: - contents: read - packages: write - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Log in to Docker Hub - uses: docker/login-action@v3 - with: - username: ${{ secrets.DOCKER_USERNAME }} - password: ${{ secrets.DOCKER_PASSWORD }} - - - name: Extract version from tag - id: meta - run: echo "version=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT - - - name: Build and push UI Docker image - uses: docker/build-push-action@v7 - with: - context: . - file: ./Dockerfile - push: true - tags: | - dxworks/scriptbee:${{ steps.meta.outputs.version }} - dxworks/scriptbee:latest - cache-from: type=gha - cache-to: type=gha,mode=max # ------------------------------------------------------------------ # BACKEND JOBS @@ -117,9 +81,9 @@ jobs: - name: Test run: dotnet test ScriptBee.slnx --no-build --verbosity normal - gateway-build-push: - name: Gateway Build & Push Docker - needs: backend-test + server-build-push: + name: Server Build & Push Docker + needs: [ ui-test, backend-test ] runs-on: ubuntu-latest permissions: contents: read @@ -140,54 +104,19 @@ jobs: - name: Extract version from tag id: meta - run: echo "version=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT + run: | + VERSION="${GITHUB_REF_NAME#v}" + echo "version=$VERSION" >> $GITHUB_OUTPUT - - name: Build and push Gateway Docker image + - name: Build and push Server Docker image uses: docker/build-push-action@v7 with: context: . file: ./Dockerfile push: true tags: | - dxworks/scriptbee/gateway:${{ steps.meta.outputs.version }} - dxworks/scriptbee/gateway:latest - cache-from: type=gha - cache-to: type=gha,mode=max - - analysis-build-push: - name: Analysis Build & Push Docker - needs: backend-test - runs-on: ubuntu-latest - permissions: - contents: read - packages: write - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Log in to Docker Hub - uses: docker/login-action@v3 - with: - username: ${{ secrets.DOCKER_USERNAME }} - password: ${{ secrets.DOCKER_PASSWORD }} - - - name: Extract version from tag - id: meta - run: echo "version=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT - - - name: Build and push Analysis Docker image - uses: docker/build-push-action@v7 - with: - context: . - file: ./analysis.Dockerfile - push: true - tags: | - dxworks/scriptbee/analysis:${{ steps.meta.outputs.version }} - dxworks/scriptbee/analysis:latest + dxworks/scriptbee:${{ steps.meta.outputs.version }} + dxworks/scriptbee:latest cache-from: type=gha cache-to: type=gha,mode=max @@ -196,7 +125,7 @@ jobs: # ------------------------------------------------------------------ create-release: name: Create GitHub Release - needs: [ ui-build-push, gateway-build-push, analysis-build-push ] + needs: [ server-build-push ] runs-on: ubuntu-latest permissions: contents: write @@ -209,3 +138,5 @@ jobs: uses: softprops/action-gh-release@v2 with: generate_release_notes: true + tag_name: ${{ github.ref_name }} + draft: true diff --git a/Dockerfile b/Dockerfile index e1c4fc8b..f83676ac 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,15 +1,16 @@ -# Build the client app +# Build the client app FROM node:24 AS build_client WORKDIR /src -COPY ./ScriptBeeClient/package*.json ./ +COPY ScriptBeeClient/package*.json ./ +RUN npm ci -RUN npm install - -COPY ./ScriptBeeClient/src ./src -COPY ./ScriptBeeClient/angular.json . -COPY ./ScriptBeeClient/tsconfig*.json ./ +COPY ScriptBeeClient/src ./src +COPY ScriptBeeClient/public ./public +COPY ScriptBeeClient/angular.json . +COPY ScriptBeeClient/tsconfig.json . +COPY ScriptBeeClient/tsconfig.app.json . RUN npm run build-prod @@ -28,6 +29,10 @@ COPY ScriptBeeWebApp/src/Workspace ScriptBeeWebApp/src/Workspace COPY ScriptBeeWebApp/src/Gateway/Application ScriptBeeWebApp/src/Gateway/Application COPY ScriptBeeWebApp/src/Gateway/Adapters ScriptBeeWebApp/src/Gateway/Adapters +COPY Plugins/ScriptRunner/DxWorks.ScriptBee.Plugin.ScriptRunner.CSharp Plugins/ScriptRunner/DxWorks.ScriptBee.Plugin.ScriptRunner.CSharp +COPY Plugins/ScriptRunner/DxWorks.ScriptBee.Plugin.ScriptRunner.Javascript Plugins/ScriptRunner/DxWorks.ScriptBee.Plugin.ScriptRunner.Javascript +COPY Plugins/ScriptRunner/DxWorks.ScriptBee.Plugin.ScriptRunner.Python Plugins/ScriptRunner/DxWorks.ScriptBee.Plugin.ScriptRunner.Python + RUN dotnet restore ScriptBeeWebApp/src/Gateway/Adapters/Web/Web.csproj RUN dotnet publish ScriptBeeWebApp/src/Gateway/Adapters/Web/Web.csproj -c Release -o publish --no-restore @@ -35,14 +40,14 @@ RUN dotnet publish ScriptBeeWebApp/src/Gateway/Adapters/Web/Web.csproj -c Releas # Build the final image FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS final +ENV ASPNETCORE_HTTP_PORTS=80 EXPOSE 80 -EXPOSE 443 WORKDIR /app COPY --from=build_webapp /app/publish . -COPY --from=build_client /src/dist/script-bee-ui /app/wwwroot +COPY --from=build_client /src/dist/script-bee-ui/browser ./wwwroot #ENV LD_LIBRARY_PATH=/app/runtimes/debian.9-x64/native/ diff --git a/DxWorks.ScriptBee.Plugin.Api/DxWorks.ScriptBee.Plugin.Api.csproj b/DxWorks.ScriptBee.Plugin.Api/DxWorks.ScriptBee.Plugin.Api.csproj index 73487d31..75e427a4 100644 --- a/DxWorks.ScriptBee.Plugin.Api/DxWorks.ScriptBee.Plugin.Api.csproj +++ b/DxWorks.ScriptBee.Plugin.Api/DxWorks.ScriptBee.Plugin.Api.csproj @@ -5,18 +5,17 @@ true DxWorks.ScriptBee.Plugin.Api This is the API for the ScriptBee plugin. - ReleaseNotes.md + $([System.IO.File]::ReadAllText("$(MSBuildProjectDirectory)/ReleaseNotes/2.0.0.md")) README.md LICENSE https://github.com/dxworks/scriptbee/tree/master/DxWorks.ScriptBee.Plugin.Api https://github.com/dxworks/scriptbee git - 1.0.0 + 2.0.0 dxworks dxworks - diff --git a/DxWorks.ScriptBee.Plugin.Api/README.md b/DxWorks.ScriptBee.Plugin.Api/README.md index 89266c19..6d3edc25 100644 --- a/DxWorks.ScriptBee.Plugin.Api/README.md +++ b/DxWorks.ScriptBee.Plugin.Api/README.md @@ -2,3 +2,76 @@ This is the API for the ScriptBee plugin. It contains the interfaces and classes that are used by the plugin to communicate with the ScriptBee service. + +For entire documentation check [ScriptBee Plugin API](https://dxworks.org/scriptbee/plugins/plugin_api.html) + +Currently, ScriptBee supports the following types of plugins: + +- [Loader Plugins](https://dxworks.org/scriptbee/plugins/loader.html) +- [Linker Plugins](https://dxworks.org/scriptbee/plugins/linker.html) +- [Helper Functions Plugins](https://dxworks.org/scriptbee/plugins/helper_functions.html) +- [Script Generator Plugins](https://dxworks.org/scriptbee/plugins/script_generator.html) +- [Script Runner Plugins](https://dxworks.org/scriptbee/plugins/script_runner.html) +- [Bundle Plugins](https://dxworks.org/scriptbee/plugins/bundle.html) + +Every plugin must have a `manifest.yaml` file in its root directory. More information about the manifest can be found in +the [manifest section](https://dxworks.org/scriptbee/plugins/manifest.html). + +## Services + +Besides the plugin interfaces, ScriptBee offers a set of services that can be used by plugins via dependency injection. + +### IHelperFunctionsContainer + +The `IHelperFunctionsContainer` service is used to register helper functions. Helper functions are a way to extend the +functionality of ScriptBee. They can be used to add custom functions that can be used in the scripts directly. + +`IHelperFunctionsContainer` wraps the helper functions and provides a way to access them. + +```csharp title="IHelperFunctionsContainer.cs" +public interface IHelperFunctionsContainer +{ + public Dictionary GetFunctionsDictionary(); + + public IEnumerable GetFunctions(); +} +``` + +[ScriptRunner](https://dxworks.org/scriptbee/plugins/script_runner.html) uses the `IHelperFunctionsContainer` service to +get the helper functions and add them +to the script engine. + +### IHelperFunctionsResultService + +The `IHelperFunctionsResultService` service is used to store the results of the helper functions and have a uniform way +to deal with script outputs. + +```csharp title="IHelperFunctionsResultService.cs" +public interface IHelperFunctionsResultService +{ + Task UploadResultAsync(string fileName, string type, string content, CancellationToken cancellationToken = default); + + Task UploadResultAsync(string fileName, string type, Stream content, CancellationToken cancellationToken = default); + + void UploadResult(string fileName, string type, string content); + + void UploadResult(string fileName, string type, Stream content); +} +``` + +[ScriptBee's default helper functions](https://github.com/dxworks/scriptbee/tree/master/Plugins/HelperFunctions/DxWorks.ScriptBee.Plugin.HelperFunctions.Default) +use the `IHelperFunctionsResultService` service to upload the results of the +helper functions. + +### Run Result Types + +ScriptBee has a set of predefined result types that can be used by the helper functions to upload the results. + +```csharp title="RunResultDefaultTypes.cs" +public static class RunResultDefaultTypes +{ + public const string ConsoleType = "Console"; + public const string FileType = "File"; + public const string RunError = "RunError"; +} +``` diff --git a/DxWorks.ScriptBee.Plugin.Api/ReleaseNotes/2.0.0.md b/DxWorks.ScriptBee.Plugin.Api/ReleaseNotes/2.0.0.md new file mode 100644 index 00000000..c638e614 --- /dev/null +++ b/DxWorks.ScriptBee.Plugin.Api/ReleaseNotes/2.0.0.md @@ -0,0 +1,6 @@ +# Version 2.0.0 + +## Changes + +- upgrade to use dotnet 10 +- update Westwind.Utilities to 5.2.8.1 diff --git a/README.md b/README.md index 9462588c..fcb8e1e1 100644 --- a/README.md +++ b/README.md @@ -16,36 +16,52 @@ The fastest way to get started with ScriptBee is using Docker Compose. ### Run with Docker -To get ScriptBee running quickly, create a `docker-compose.yaml` file with the following content and run -`docker compose up -d`: +The fastest way to get started is by using Docker Compose. Create a `docker-compose.yaml` file with the following +configuration and run `docker compose up -d`: ```yaml -version: "3.8" services: mongo: - image: mongo:4.4 + image: mongo:8.0.4 container_name: mongo restart: unless-stopped + environment: + MONGO_INITDB_ROOT_USERNAME: root + MONGO_INITDB_ROOT_PASSWORD: example + ports: + - "27017:27017" volumes: - mongodb_data:/data/db scriptbee: - image: dxworks/scriptbee + image: dxworks/scriptbee:latest + user: root ports: - "4201:80" volumes: - - scriptbee_data:/root/.scriptbee + - ./database/scriptbee:/root/.scriptbee + - /var/run/docker.sock:/var/run/docker.sock environment: - - ConnectionStrings__mongodb=mongodb://mongo:27017/ScriptBee?authSource=admin + - ConnectionStrings__mongodb=mongodb://root:example@mongo:27017/ScriptBee?authSource=admin - UserFolder__UserFolderPath=/root/.scriptbee + - SCRIPTBEE__ANALYSIS__DRIVER=docker + - SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET=unix:///var/run/docker.sock + - SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH=${PWD}/database/scriptbee depends_on: - mongo volumes: mongodb_data: - scriptbee_data: ``` +#### Docker Hosting Tips + +| Setting | Importance | +|:----------------------------------|:---------------------------------------------------------------------------------| +| **`user: root`** | Allows the ScriptBee container to communicate with the host's Docker engine. | +| **`${PWD}`** | Ensures absolute path resolution on the host machine for analysis volume mounts. | +| **`unix:///var/run/docker.sock`** | The standard socket for Docker-out-of-Docker communication. | + ### Accessing the UI Once the containers are running, open your browser and navigate to **[http://localhost:4201](http://localhost:4201)**. @@ -62,8 +78,8 @@ ScriptBee can be configured using environment variables. Below are the most impo | `UserFolder__UserFolderPath` | Host path for storing project data and shared files. | | | `SCRIPTBEE_ANALYSIS__DRIVER` | How analysis instances are managed: `docker` or `kubernetes`. | `docker` | -For more detailed information on advanced configuration, check the * -*[Deployment Features](docs/architecture/features.md)** documentation. +For more detailed information on advanced configuration, check the +*[Deployment Features](docs/architecture/features.md)* documentation. --- @@ -73,5 +89,5 @@ Full documentation is available at [https://dxworks.org/scriptbee/](https://dxwo ## Contributing -For information on the repository structure, development setup, and how to contribute, please refer to our * -*[Contributing Guide](.github/CONTRIBUTING.md)**. +For information on the repository structure, development setup, and how to contribute, please refer to our +*[Contributing Guide](.github/CONTRIBUTING.md)*. diff --git a/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/AnalysisInstanceDockerAdapter.cs b/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/AnalysisInstanceDockerAdapter.cs index ce969428..e04f4f30 100644 --- a/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/AnalysisInstanceDockerAdapter.cs +++ b/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/AnalysisInstanceDockerAdapter.cs @@ -81,6 +81,7 @@ await client.Containers.StartContainerAsync( client, response.ID, analysisDockerConfig.Network, + analysisDockerConfig.Port, hostPort, cancellationToken ); @@ -204,6 +205,7 @@ private List GetEnvironmentVariables( $"ScriptBee__ProjectId={projectDetails.Id}", $"ScriptBee__ProjectName={projectDetails.Name}", $"ConnectionStrings__mongodb={mongoDbConnectionString}", + $"ASPNETCORE_HTTP_PORTS={config.Value.Port}", }; if ( @@ -257,7 +259,8 @@ private static async Task GetContainerUrl( DockerClient client, string containerId, string? networkName, - int port, + int internalPort, + int hostPort, CancellationToken cancellationToken ) { @@ -268,12 +271,12 @@ CancellationToken cancellationToken if (networkName == null) { - return $"http://localhost:{port}"; + return $"http://localhost:{hostPort}"; } return containerInfo.NetworkSettings.Networks.TryGetValue(networkName, out var network) - ? $"http://{network.IPAddress}:{port}" - : $"http://localhost:{port}"; + ? $"http://{network.IPAddress}:{internalPort}" + : $"http://localhost:{hostPort}"; } private async Task PullImageIfNeeded( diff --git a/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/Config/AnalysisDockerConfig.cs b/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/Config/AnalysisDockerConfig.cs index 093dd12f..65968ef4 100644 --- a/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/Config/AnalysisDockerConfig.cs +++ b/ScriptBeeWebApp/src/Gateway/Adapters/Analysis.Instance.Docker/Config/AnalysisDockerConfig.cs @@ -1,10 +1,10 @@ -namespace ScriptBee.Analysis.Instance.Docker.Config; +namespace ScriptBee.Analysis.Instance.Docker.Config; public class AnalysisDockerConfig { public required string DockerSocket { get; init; } - public int Port { get; init; } = 8080; + public int Port { get; init; } = 80; public string? Network { get; init; } diff --git a/ScriptBeeWebApp/src/Gateway/Adapters/Web/Extensions/ScriptBeeAnalysisConfigExtensions.cs b/ScriptBeeWebApp/src/Gateway/Adapters/Web/Extensions/ScriptBeeAnalysisConfigExtensions.cs index 07c94b08..6cec9f1f 100644 --- a/ScriptBeeWebApp/src/Gateway/Adapters/Web/Extensions/ScriptBeeAnalysisConfigExtensions.cs +++ b/ScriptBeeWebApp/src/Gateway/Adapters/Web/Extensions/ScriptBeeAnalysisConfigExtensions.cs @@ -18,7 +18,12 @@ ConfigurationManager configurationManager .GetSection("ScriptBee:Analysis") .Get()!; - if (scriptBeeAnalysisConfig.Driver != "Docker") + if ( + !scriptBeeAnalysisConfig.Driver.Equals( + "Docker", + StringComparison.InvariantCultureIgnoreCase + ) + ) { throw new AnalysisInstanceDriverTypeNotSupported(scriptBeeAnalysisConfig.Driver); } diff --git a/ScriptBeeWebApp/src/Gateway/Adapters/Web/Program.cs b/ScriptBeeWebApp/src/Gateway/Adapters/Web/Program.cs index a0574d67..9a672770 100644 --- a/ScriptBeeWebApp/src/Gateway/Adapters/Web/Program.cs +++ b/ScriptBeeWebApp/src/Gateway/Adapters/Web/Program.cs @@ -1,4 +1,4 @@ -using FluentValidation; +using FluentValidation; using Microsoft.AspNetCore.Server.Kestrel.Core; using ScriptBee.Artifacts.Extensions; using ScriptBee.Common.Web; @@ -51,6 +51,7 @@ app.MapSwaggerUi(); } +app.UseDefaultFiles(); app.UseStaticFiles(); app.UseRouting(); @@ -62,20 +63,10 @@ app.UseEndpoints(_ => { }); -// app.UseSpa(spa => -// { -// spa.Options.SourcePath = "../ScriptBeeClient"; -// -// if (app.Environment.IsDevelopment()) -// { -// spa.UseProxyToSpaDevelopmentServer("http://localhost:4200"); -// } -// }); - -// app.MapHub("/api/fileWatcherHub"); - app.UseEndpointDefinitions(); +app.MapFallbackToFile("index.html"); + app.Run(); public partial class Program; diff --git a/ScriptBeeWebApp/test/Gateway/Adapters/Analysis.Instance.Docker.Tests/AnalysisInstanceDockerAdapterTest.cs b/ScriptBeeWebApp/test/Gateway/Adapters/Analysis.Instance.Docker.Tests/AnalysisInstanceDockerAdapterTest.cs index 49b85511..59dfc5b0 100644 --- a/ScriptBeeWebApp/test/Gateway/Adapters/Analysis.Instance.Docker.Tests/AnalysisInstanceDockerAdapterTest.cs +++ b/ScriptBeeWebApp/test/Gateway/Adapters/Analysis.Instance.Docker.Tests/AnalysisInstanceDockerAdapterTest.cs @@ -92,7 +92,7 @@ public async Task Allocate_ShouldCreateAndStartContainerAndReturnUrlWithNetworkI // Assert containerUrl.ShouldStartWith("http://"); - containerUrl.ShouldContain($":{_testPort}"); + containerUrl.ShouldContain($":{_configOptions.Value.Port}"); var containers = await _dockerFixture.DockerClient.Containers.ListContainersAsync( new ContainersListParameters { All = true }, TestContext.Current.CancellationToken @@ -107,7 +107,7 @@ public async Task Allocate_ShouldCreateAndStartContainerAndReturnUrlWithNetworkI .NetworkSettings.Networks[DockerFixture.TestNetworkName] .IPAddress.ShouldNotBeNullOrEmpty(); containerUrl.ShouldBe( - $"http://{ourContainer.NetworkSettings.Networks[DockerFixture.TestNetworkName].IPAddress}:{_testPort}" + $"http://{ourContainer.NetworkSettings.Networks[DockerFixture.TestNetworkName].IPAddress}:{_configOptions.Value.Port}" ); } @@ -157,6 +157,39 @@ await adapter.Allocate( containers.First().NetworkSettings.Networks.ShouldContainKey(DockerFixture.TestNetworkName); } + [Fact] + public async Task Allocate_ShouldReturnLocalhostWithHostPort_WhenNoNetworkIsConfigured() + { + // Arrange + var config = new AnalysisDockerConfig + { + DockerSocket = _dockerFixture.DockerClient.Configuration.EndpointBaseUri.ToString(), + Network = null, + UserFolderVolumePath = "/root/.scriptbee", + }; + var adapter = new AnalysisInstanceDockerAdapter( + Options.Create(config), + _userFolderOptions, + _configuration, + _logger, + _freePortProvider + ); + var projectDetails = ProjectDetailsFixture.BasicProjectDetails(ProjectId.FromValue("id")); + var instanceId = new InstanceId(Guid.NewGuid()); + var image = new AnalysisInstanceImage(DockerFixture.TestImageName); + + // Act + var containerUrl = await adapter.Allocate( + projectDetails, + instanceId, + image, + TestContext.Current.CancellationToken + ); + + // Assert + containerUrl.ShouldBe($"http://localhost:{_testPort}"); + } + [Fact] public async Task Allocate_ShouldPassEnvironmentVariables() { @@ -200,6 +233,9 @@ await adapter.Allocate( containerInspect.Config.Env.ShouldContain( $"UserFolder__UserFolderPath={_configOptions.Value.UserFolderVolumePath}" ); + containerInspect.Config.Env.ShouldContain( + $"ASPNETCORE_HTTP_PORTS={_configOptions.Value.Port}" + ); } [Fact] diff --git a/analysis.Dockerfile b/analysis.Dockerfile index 02c1b489..b7e31860 100644 --- a/analysis.Dockerfile +++ b/analysis.Dockerfile @@ -1,4 +1,4 @@ -FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build_webapp +FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build_webapp WORKDIR /app @@ -19,8 +19,8 @@ RUN dotnet publish ScriptBeeWebApp/src/Analysis/Adapters/Analysis.Web/Analysis.W # Build the final image FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS final +ENV ASPNETCORE_HTTP_PORTS=80 EXPOSE 80 -EXPOSE 443 WORKDIR /app diff --git a/docker-compose.yaml b/docker-compose.yaml index c63334db..31b54d2b 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -1,22 +1,34 @@ -version: "3.8" services: mongo: - image: mongo:4.4 + image: mongo:8.0.4 container_name: mongo restart: unless-stopped + environment: + MONGO_INITDB_ROOT_USERNAME: root + MONGO_INITDB_ROOT_PASSWORD: example + ports: + - "27017:27017" volumes: - - ${MONGODB_DATA}:/data/db + - mongodb_data:/data/db - - scripbee: - image: dxworks/scriptbee + scriptbee: + image: dxworks/scriptbee:latest + user: root ports: - "4201:80" volumes: - - ${SCRIPTBEE_DATA}:/root/.scriptbee + - ./database/scriptbee:/root/.scriptbee + - /var/run/docker.sock:/var/run/docker.sock environment: - - ConnectionStrings__mongodb=mongodb://mongo:27017/ScriptBee?authSource=admin - - UserFolder__UserFolderPath=${SCRIPTBEE_DATA} + - ConnectionStrings__mongodb=mongodb://root:example@mongo:27017/ScriptBee?authSource=admin + - UserFolder__UserFolderPath=/root/.scriptbee + - SCRIPTBEE__ANALYSIS__DRIVER=docker + - SCRIPTBEE__ANALYSIS__DOCKER__NETWORK=scriptbee_default + - SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET=unix:///var/run/docker.sock + - SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH=${PWD}/database/scriptbee depends_on: - mongo + +volumes: + mongodb_data: diff --git a/docs/home/installation.md b/docs/home/installation.md index 40ba655e..529c655b 100644 --- a/docs/home/installation.md +++ b/docs/home/installation.md @@ -2,9 +2,9 @@ ## Prerequisites -- Mongo 4.4 -- Dotnet 6.0 -- Node 16.0 +- Mongo 8.0.4 +- Dotnet 10.0 +- Node 24.0 ## Docker Compose @@ -14,25 +14,34 @@ an environment variable. An example of the `docker-compose.yaml`: ```yaml title="docker-compose.yaml" -version: '3.8' services: mongo: - image: mongo:4.4 + image: mongo:8.0.4 container_name: mongo restart: unless-stopped + environment: + MONGO_INITDB_ROOT_USERNAME: root + MONGO_INITDB_ROOT_PASSWORD: example + ports: + - "27017:27017" volumes: - ./database:/data/db - scripbee: - image: dxworks/scriptbee + scriptbee: + image: dxworks/scriptbee:latest + user: root ports: - '4201:80' volumes: - - /host/scriptbee_data:/root/.scriptbee + - ./database/scriptbee:/root/.scriptbee + - /var/run/docker.sock:/var/run/docker.sock environment: - - UserFolder__UserFolderPath=/host/scriptbee_data - - ConnectionStrings__mongodb=mongodb://mongo:27017/ScriptBee?authSource=admin + - ConnectionStrings__mongodb=mongodb://root:example@mongo:27017/ScriptBee?authSource=admin + - UserFolder__UserFolderPath=/root/.scriptbee + - SCRIPTBEE__ANALYSIS__DRIVER=docker + - SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET=unix:///var/run/docker.sock + - SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH=${PWD}/database/scriptbee depends_on: - mongo ``` @@ -45,29 +54,26 @@ To run ScriptBee simply run the following command: docker-compose up ``` -### Driver Configuration - -The Analysis service can be configured to use different drivers for managing analysis instances. Currently, the primary driver is **Docker**. +### Configuration -#### Docker Driver Configuration +ScriptBee is configured primarily through environment variables. The most critical settings relate to how it manages analysis instances via Docker. -To use the Docker driver, set the following environment variable: +### Driver Configuration +To enable Docker-managed analysis, set: ```dotenv SCRIPTBEE__ANALYSIS__DRIVER=docker ``` -Once enabled, the Gateway will spawn analysis instances as standalone Docker containers. - -##### Minimum Configuration +#### Minimum Configuration -For a basic setup (e.g., using the default Docker Compose), the minimum environment variables required are: +For a basic setup, the minimum required environment variables are: ```yaml environment: - SCRIPTBEE__ANALYSIS__DRIVER=docker - - SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET=unix:///var/run/docker.sock # or npipe://./pipe/docker_engine on Windows - - SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH=/path/to/your/data # Absolute path to SCRIPTBEE_DATA on host + - SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET=unix:///var/run/docker.sock + - SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH=${PWD}/database/scriptbee ``` ##### Full Configuration Reference @@ -78,11 +84,11 @@ You can fine-tune the Docker driver behavior using the following environment var | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | ----------------------------------- | | `SCRIPTBEE__ANALYSIS__IMAGE` | The Docker image used for analysis instances. | `dxworks/scriptbee/analysis:latest` | | `SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET` | URI to the Docker daemon socket. | `unix:///var/run/docker.sock` | -| `SCRIPTBEE__ANALYSIS__DOCKER__PORT` | The internal port the analysis container listens on. | `8080` | +| `SCRIPTBEE__ANALYSIS__DOCKER__PORT` | The internal port the analysis container listens on. | `80` | | `SCRIPTBEE__ANALYSIS__DOCKER__NETWORK` | The Docker network name to attach containers to (required if using a custom network for MongoDB). | _None_ | | `SCRIPTBEE__ANALYSIS__DOCKER__MONGODBCONNECTIONSTRING` | MongoDB connection string for the analysis instance. | _Inherits Gateway's if omitted_ | | `SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERVOLUMEPATH` | The mount point for project data _inside_ the analysis container. | `/root/.scriptbee` | -| `SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH` | THE absolute path on the **host machine** where project data is stored. | _Inherits Gateway's if omitted_ | +| `SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH` | THE absolute path on the **host machine** where project data is stored. | `${PWD}/database/scriptbee` | > [!IMPORTANT] > The `SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH` must be an **absolute path** on your host machine. This is because the Gateway tells the Docker daemon to mount this path into the new analysis containers. If this is incorrect, the analysis service will not be able to find your scripts or models. @@ -97,7 +103,7 @@ For a basic setup (e.g., using the default Docker Compose), the minimum environm environment: - SCRIPTBEE__ANALYSIS__DRIVER=docker - SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET=unix:///var/run/docker.sock # or npipe://./pipe/docker_engine on Windows - - SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH=/path/to/your/data # Absolute path to SCRIPTBEE_DATA on host + - SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERHOSTPATH=${PWD}/database/scriptbee # Absolute path to SCRIPTBEE_DATA on host ``` ##### Full Configuration Reference @@ -108,7 +114,7 @@ You can fine-tune the Docker driver behavior using the following environment var | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | ----------------------------------- | | `SCRIPTBEE__ANALYSIS__IMAGE` | The Docker image used for analysis instances. | `dxworks/scriptbee/analysis:latest` | | `SCRIPTBEE__ANALYSIS__DOCKER__DOCKERSOCKET` | URI to the Docker daemon socket. | `unix:///var/run/docker.sock` | -| `SCRIPTBEE__ANALYSIS__DOCKER__PORT` | The internal port the analysis container listens on. | `8080` | +| `SCRIPTBEE__ANALYSIS__DOCKER__PORT` | The internal port the analysis container listens on. | `80` | | `SCRIPTBEE__ANALYSIS__DOCKER__NETWORK` | The Docker network name to attach containers to (required if using a custom network for MongoDB). | _None_ | | `SCRIPTBEE__ANALYSIS__DOCKER__MONGODBCONNECTIONSTRING` | MongoDB connection string for the analysis instance. | _Inherits Gateway's if omitted_ | | `SCRIPTBEE__ANALYSIS__DOCKER__USERFOLDERVOLUMEPATH` | The mount point for project data _inside_ the analysis container. | `/root/.scriptbee` | @@ -165,16 +171,26 @@ following environment variable: `UserFolder__UserFolderPath`. This should be the absolute path of the host machine to the folder where the ScriptBee data is stored. ```yaml -scripbee: - image: dxworks/scriptbee +scriptbee: + image: dxworks/scriptbee:latest + user: root volumes: - - ./scriptbee_data:/root/.scriptbee + - ./database/scriptbee:/root/.scriptbee environment: - - UserFolder__UserFolderPath=/root/scriptbee_data + - UserFolder__UserFolderPath=/root/.scriptbee ``` + +### Docker Hosting Tips + +When hosting ScriptBee with Docker, pay attention to the following settings: + +- **`user: root`**: Required for the main container to manage other containers (Analysis instances). +- **`${PWD}`**: Always use `${PWD}` or an absolute path for `USERFOLDERHOSTPATH`. Docker requires absolute paths when creating containers from within another container. +- **`unix:///var/run/docker.sock`**: Standard socket for Docker communication. + ## Kubernetes ### Driver Configuration