From 28a4ea68aac69614bdcde76417ebf4fe92727ba5 Mon Sep 17 00:00:00 2001 From: CoJaques Date: Sun, 17 Aug 2025 10:50:20 +0200 Subject: [PATCH 1/3] feat: delete unused readme --- src/DataAggregator.Processor/README.md | 201 ------------------------- 1 file changed, 201 deletions(-) delete mode 100644 src/DataAggregator.Processor/README.md diff --git a/src/DataAggregator.Processor/README.md b/src/DataAggregator.Processor/README.md deleted file mode 100644 index 995a406..0000000 --- a/src/DataAggregator.Processor/README.md +++ /dev/null @@ -1,201 +0,0 @@ -(Simple README, generated with a LLM) - -# DataAggregator.Processor - -Prediction service for the DataAggregator project that enriches collected data with predictions based on ONNX models. - -## Features - -- **Real-time predictions** : Executes ONNX models to predict values based on sensor data -- **Per-machine configuration** : Each machine can have its own model and configuration -- **InfluxDB integration** : Stores predictions in the same database as raw data -- **Robust error handling** : Automatically stops predictions for a machine in case of error -- **Health checks** : Monitoring endpoints to verify service status -- **Connection optimization** : Reuses InfluxDB connections to avoid unnecessary reinitializations -- **Preprocessing strategies** : Strategy pattern for different data preparation methods - -## Configuration - -### appsettings.json - -```json -{ - "PredictionService": { - "RegistrationServiceUrl": "http://localhost:5001", - "GlobalCycleIntervalSeconds": 30, - "Machines": [ - { - "MachineName": "OpenCN-Machine-001", - "Enabled": true, - "ModelPath": "/app/models/opencn-machine-001.onnx", - "PreprocessingStrategy": "ActuatorCurrentFeatureExtractor", - "InputSensors": ["V1", "V2", "V3", "V4", "V5", "V6"], - "PredictionSensorName": "PredictedActuatorCurrent", - "WindowSizeSeconds": 60, - "CycleIntervalSeconds": 30 - } - ] - } -} -``` - -### Configuration parameters - -- **MachineName** : Machine name (must match the one registered in the Registration Service) -- **Enabled** : Enables/disables predictions for this machine -- **ModelPath** : Path to the ONNX model file (in Docker volume `/app/models`) -- **PreprocessingStrategy** : Name of the preprocessing strategy to use -- **InputSensors** : List of input sensors required by the model -- **PredictionSensorName** : Name of the prediction sensor (will be stored in InfluxDB) -- **WindowSizeSeconds** : Data window size in seconds -- **CycleIntervalSeconds** : Prediction execution frequency in seconds - -## Usage with Docker - -### 1. Prepare ONNX models - -Place your ONNX models in the `models/` folder at the project root: - -``` -models/ -├── opencn-machine-001.onnx -├── machine-002.onnx -└── ... -``` - -### 2. Launch the service - -```bash -# Launch all services -docker-compose up - -# Launch only the prediction service -docker-compose up processor -``` - -### 3. Check service status - -```bash -# Health check -curl http://localhost:5002/api/healthcheck - -# Swagger UI (in development) -http://localhost:5002/swagger -``` - -## Architecture - -### Main components - -1. **PredictionBackgroundService** : Background service that manages predictions for all machines -2. **MachinePredictionProcessor** : Processes predictions for a specific machine -3. **OnnxPredictionEngine** : ONNX prediction engine with model caching -4. **InfluxV3Repository** : Interface with InfluxDB for reading/writing data -5. **RegistrationServiceClient** : Client to retrieve machine information -6. **PreprocessingStrategyFactory** : Factory to create preprocessing strategies -7. **ActuatorCurrentFeatureExtractor** : Preprocessing strategy for current sensors - -### Data flow - -1. The service retrieves machine configuration from `appsettings.json` -2. For each enabled machine, it retrieves its information from the Registration Service -3. It reads a data window from InfluxDB for input sensors -4. It prepares data using the configured preprocessing strategy -5. It executes the prediction with the ONNX model -6. It stores the result in InfluxDB with the "Prediction" tag - -### Optimizations - -- **InfluxDB connection cache** : Avoids unnecessary client reinitializations -- **ONNX model cache** : Loads models only once in memory -- **Error handling** : Continues processing even if one machine fails - -## Error handling - -- **Missing model** : Service stops at startup if an ONNX model is not found -- **Insufficient data** : Predictions are stopped for a machine if input data doesn't match expected format -- **Prediction errors** : Errors are logged and predictions are temporarily stopped for the concerned machine -- **Connection issues** : Automatic InfluxDB reconnection handling - -## Monitoring - -### Logs - -The service uses Serilog for logging with the following levels: -- **Information** : Service start/stop, successful predictions -- **Warning** : Missing data, invalid configurations -- **Error** : Prediction errors, connection issues -- **Debug** : Operation details (in development mode) - -### Health Check - -The `/api/healthcheck` endpoint returns service status: -- **200 OK** : Service functional -- **500 Internal Server Error** : Service issue - -## Development - -### Project structure - -``` -src/DataAggregator.Processor/ -├── Configuration/ -│ ├── MachinePredictionConfig.cs -│ └── PredictionServiceConfiguration.cs -├── Controllers/ -│ └── HealthCheckController.cs -├── Services/ -│ ├── DataAccess/ -│ │ ├── IInfluxV3Repository.cs -│ │ ├── InfluxV3Repository.cs -│ │ ├── IRegistrationServiceClient.cs -│ │ └── RegistrationServiceClient.cs -│ ├── Prediction/ -│ │ ├── IOnnxPredictionEngine.cs -│ │ ├── OnnxPredictionEngine.cs -│ │ └── MachinePredictionProcessor.cs -│ ├── Preprocessing/ -│ │ ├── IPreprocessingStrategy.cs -│ │ ├── IPreprocessingStrategyFactory.cs -│ │ ├── PreprocessingStrategyFactory.cs -│ │ ├── ActuatorCurrentFeatureExtractor.cs -│ │ └── MathUtils.cs -│ └── Background/ -│ └── PredictionBackgroundService.cs -├── appsettings.json -├── appsettings.Development.json -├── Program.cs -└── Dockerfile -``` - -### Testing - -To test the service locally: - -1. Ensure Registration and InfluxDB services are started -2. Place a test ONNX model in the `models/` folder -3. Configure `appsettings.json` with correct values -4. Launch the service: `dotnet run` - -## Preprocessing strategies - -### ActuatorCurrentFeatureExtractor - -This strategy extracts 14 features from current sensor data: - -1. **Statistical features** : Mean, standard deviation, min, max, etc. -2. **Correlation features** : Correlation between axes -3. **Activity features** : Global activity ratio -4. **Stability features** : Temporal stability of signals - -Features are normalized with Z-score and formatted for ONNX model input. - -## Integration with DataAggregator ecosystem - -The prediction service integrates perfectly with the existing architecture: - -- **Reuses** shared data models (`IMeasurementData`, `DeviceRegistrationResponse`) -- **Follows** the same configuration and logging patterns -- **Uses** the same technologies (Serilog, ASP.NET Core, Docker) -- **Integrates** with existing services without modifying them -- **Optimizes** performance with caches and reused connections \ No newline at end of file From bfb731fe46b0c3ec02ae8b1aa4a2f064c94d62a2 Mon Sep 17 00:00:00 2001 From: Colin Jaques <91125307+CoJaques@users.noreply.github.com> Date: Thu, 26 Mar 2026 21:15:54 +0100 Subject: [PATCH 2/3] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index e3aff2c..4740319 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,10 @@ **DataAggregator** is a modular infrastructure for collecting, registering, and processing industrial data. It provides a flexible backbone to acquire machine signals, store them in time-series and relational databases, and run ML-based processing pipelines. +It is designed to scale from a single machine to an entire plant, handling heterogeneous environments through dedicated collectors for each data source. Data can be visualized in real time using tools like Grafana or any custom frontend, while processing can run either at the edge or centrally. This enables the application of custom algorithms or ONNX-based models directly on the collected signals. + +[demo.webm](https://github.com/user-attachments/assets/730faf7f-43d0-4bb7-9f42-046890b8afd2) + example-architecture --- From e59d20b377601db609df8b439179b9ad625d8f79 Mon Sep 17 00:00:00 2001 From: CoJaques Date: Thu, 26 Mar 2026 21:27:47 +0100 Subject: [PATCH 3/3] feat: add license and update readme --- LICENSE.txt | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 47 +++++++++++-------- 2 files changed, 161 insertions(+), 19 deletions(-) create mode 100644 LICENSE.txt diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..973c104 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,133 @@ +Required Notice: Copyright Colin Jaques (https://github.com/CoJaques) + +# PolyForm Noncommercial License 1.0.0 + + + +## Acceptance + +In order to get any license under these terms, you must agree +to them as both strict obligations and conditions to all +your licenses. + +## Copyright License + +The licensor grants you a copyright license for the +software to do everything you might do with the software +that would otherwise infringe the licensor's copyright +in it for any permitted purpose. However, you may +only distribute the software according to [Distribution +License](#distribution-license) and make changes or new works +based on the software according to [Changes and New Works +License](#changes-and-new-works-license). + +## Distribution License + +The licensor grants you an additional copyright license +to distribute copies of the software. Your license +to distribute covers distributing the software with +changes and new works permitted by [Changes and New Works +License](#changes-and-new-works-license). + +## Notices + +You must ensure that anyone who gets a copy of any part of +the software from you also gets a copy of these terms or the +URL for them above, as well as copies of any plain-text lines +beginning with `Required Notice:` that the licensor provided +with the software. For example: + +> Required Notice: Copyright Yoyodyne, Inc. (http://example.com) + +## Changes and New Works License + +The licensor grants you an additional copyright license to +make changes and new works based on the software for any +permitted purpose. + +## Patent License + +The licensor grants you a patent license for the software that +covers patent claims the licensor can license, or becomes able +to license, that you would infringe by using the software. + +## Noncommercial Purposes + +Any noncommercial purpose is a permitted purpose. + +## Personal Uses + +Personal use for research, experiment, and testing for +the benefit of public knowledge, personal study, private +entertainment, hobby projects, amateur pursuits, or religious +observance, without any anticipated commercial application, +is use for a permitted purpose. + +## Noncommercial Organizations + +Use by any charitable organization, educational institution, +public research organization, public safety or health +organization, environmental protection organization, +or government institution is use for a permitted purpose +regardless of the source of funding or obligations resulting +from the funding. + +## Fair Use + +You may have "fair use" rights for the software under the +law. These terms do not limit them. + +## No Other Rights + +These terms do not allow you to sublicense or transfer any of +your licenses to anyone else, or prevent the licensor from +granting licenses to anyone else. These terms do not imply +any other licenses. + +## Patent Defense + +If you make any written claim that the software infringes or +contributes to infringement of any patent, your patent license +for the software granted under these terms ends immediately. If +your company makes such a claim, your patent license ends +immediately for work on behalf of your company. + +## Violations + +The first time you are notified in writing that you have +violated any of these terms, or done anything with the software +not covered by your licenses, your licenses can nonetheless +continue if you come into full compliance with these terms, +and take practical steps to correct past violations, within +32 days of receiving notice. Otherwise, all your licenses +end immediately. + +## No Liability + +***As far as the law allows, the software comes as is, without +any warranty or condition, and the licensor will not be liable +to you for any damages arising out of these terms or the use +or nature of the software, under any kind of legal claim.*** + +## Definitions + +The **licensor** is the individual or entity offering these +terms, and the **software** is the software the licensor makes +available under these terms. + +**You** refers to the individual or entity agreeing to these +terms. + +**Your company** is any legal entity, sole proprietorship, +or other kind of organization that you work for, plus all +organizations that have control over, are under the control of, +or are under common control with that organization. **Control** +means ownership of substantially all the assets of an entity, +or the power to direct its management and policies by vote, +contract, or otherwise. Control can be direct or indirect. + +**Your licenses** are all the licenses granted to you for the +software under these terms. + +**Use** means anything you do with the software requiring one +of your licenses. diff --git a/README.md b/README.md index e3aff2c..b41b52a 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,11 @@ # 🚀 DataAggregator -[![.NET 9](https://img.shields.io/badge/.NET-9.0-blueviolet?logo=dotnet&logoColor=white)](https://dotnet.microsoft.com/) -[![Docker](https://img.shields.io/badge/Docker-ready-2496ED?logo=docker&logoColor=white)](https://www.docker.com/) -[![InfluxDB 3](https://img.shields.io/badge/InfluxDB-3.0-22ADF6?logo=influxdb&logoColor=white)](https://www.influxdata.com/) -[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-15-336791?logo=postgresql&logoColor=white)](https://www.postgresql.org/) +[![.NET 9](https://img.shields.io/badge/.NET-9.0-blueviolet?logo=dotnet&logoColor=white)](https://dotnet.microsoft.com/) +[![Docker](https://img.shields.io/badge/Docker-ready-2496ED?logo=docker&logoColor=white)](https://www.docker.com/) +[![InfluxDB 3](https://img.shields.io/badge/InfluxDB-3.0-22ADF6?logo=influxdb&logoColor=white)](https://www.influxdata.com/) +[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-15-336791?logo=postgresql&logoColor=white)](https://www.postgresql.org/) -**DataAggregator** is a modular infrastructure for collecting, registering, and processing industrial data. +**DataAggregator** is a modular infrastructure for collecting, registering, and processing industrial data. It provides a flexible backbone to acquire machine signals, store them in time-series and relational databases, and run ML-based processing pipelines. example-architecture @@ -37,7 +37,7 @@ DataAggregator/ │ ├─ DataAggregator.Collector.FileCollector/ # File-based collector for demo/testing │ ├─ DataAggregator.Registration/ # Registration service (ASP.NET Core) │ └─ DataAggregator.Processor/ # Processing service (ONNX models, ML logic) -├─ tests/ +├─ tests/ │ ├─ DataAggregator.Registration.Tests/ # Unit tests for Registration │ ├─ DataAggregator.Collector.Tests/ # Unit tests for Registration │ └─ DataAggregator.Processor.Tests/ # Unit tests for Processor @@ -47,13 +47,13 @@ DataAggregator/ ## 🌐 Related Repositories -- 🔧 **Deployment example (full stack with configs, Grafana, etc.)** +- 🔧 **Deployment example (full stack with configs, Grafana, etc.)** 👉 [DataAggregator-demo](https://github.com/CoJaques/DataAggregator-demo) -- 📊 **Model training & analysis pipeline** +- 📊 **Model training & analysis pipeline** 👉 [Data-analysis](https://github.com/CoJaques/Data-analysis) -- 🛠️ **OpenCN fork with connector support** (work in progress) +- 🛠️ **OpenCN fork with connector support** (work in progress) 👉 [OpenCN branch `cmctl-bufferization`](https://gitlab.com/CoJaques/opencn/-/tree/cmctl-bufferization?ref_type=heads) --- @@ -62,10 +62,10 @@ DataAggregator/ - **.NET 9** → Backend services (Collector, Registration, Processor) - **ML.NET** -> Data processing -- **Docker / Docker Compose** → Deployment & orchestration -- **InfluxDB v3** → Time-series data storage -- **PostgreSQL** → Relational storage & metadata -- **Grafana** (in demo repo) → Data visualization +- **Docker / Docker Compose** → Deployment & orchestration +- **InfluxDB v3** → Time-series data storage +- **PostgreSQL** → Relational storage & metadata +- **Grafana** (in demo repo) → Data visualization --- @@ -73,20 +73,29 @@ DataAggregator/ DataAggregator is built around a **modular and distributed architecture**: -- **Collector** → Acquires machine signals (e.g., currents from CNC drives) via OpenCN or file input. -- **Registration** → Manages device registration and configuration in PostgreSQL. -- **Processor** → Extracts features and runs ML inference (ONNX models) to classify machine states. -- **Databases** → InfluxDB v3 stores raw and processed signals; PostgreSQL stores device metadata. +- **Collector** → Acquires machine signals (e.g., currents from CNC drives) via OpenCN or file input. +- **Registration** → Manages device registration and configuration in PostgreSQL. +- **Processor** → Extracts features and runs ML inference (ONNX models) to classify machine states. +- **Databases** → InfluxDB v3 stores raw and processed signals; PostgreSQL stores device metadata. -This design ensures scalability across multiple machines and flexibility for integration with existing industrial setups. +This design ensures scalability across multiple machines and flexibility for integration with existing industrial setups. --- ## 🚀 Getting Started -👉 For a ready-to-use deployment example (including Docker setup, Grafana dashboards, and configs), check out: +👉 For a ready-to-use deployment example (including Docker setup, Grafana dashboards, and configs), check out: 📦 [**DataAggregator-demo**](https://github.com/CoJaques/DataAggregator-demo) +--- +## 📄 License + +This project is licensed under the PolyForm Noncommercial License 1.0.0. +See the [LICENSE](./LICENSE.txt) file for details. + +This project was developed as part of a Bachelor thesis and is published with permission. +Any commercial use is not permitted without prior discussion with the author. + --- `this readme was enhanced by AI to clean the formatting and structure the content`