Conversation
- Add collapsible sidebar with smooth transitions - Implement hierarchical tree view with folder/file icons - Add responsive design for collapsed/expanded states - Add custom scrollbars and improved button interactions - Fix TailwindCSS configuration and remove unused packages - Consolidate .gitignore files - Add comprehensive README with setup instructions Dashboard now has a fully functional sidebar with: - Expandable tree structure - Collapsible sidebar with icon-only view - Modern gradient header design - Responsive layout adjustments
…n logging Major Features: - Generic Zarr store structure support (any hierarchy) - Real-time S3 connection with status indicators - Accordion-style variable data viewer - YAML-configurable endpoints - Root/nested variable navigation - NaN value JSON serialization - Comprehensive error handling Technical Improvements: - Clean separation of S3Service and UI components - Zarr-native path handling (removed artificial root prefix) - Code cleanup and refactoring - Multi-store testing capability - Professional logging output - Updated documentation and gitignore Backend Changes: - Generic S3Service with zarr_fuse integration - Robust error handling and fallback mechanisms - YAML-based endpoint management - Clean log messages without visual clutter Frontend Changes: - Complete UI overhaul for generic store support - Accordion-style variable display - Real-time status indicators with progress bar - Clickable store navigation - Error resilient design Configuration: - Multiple S3 bucket support via env.example - Flexible YAML endpoint configuration - Clean development environment setup
Backend Add routers/logs.py with GET /api/logs?limit= returning only error/warning entries from latest file in log_store.zarr/logs. Add LOGS_DIR to core/config.py. Wire router in backend/main.py. Frontend Update LogPanel.tsx to fetch from /api/logs, remove All/Store filters and mock data, show “Backend” badge and “No logs yet” empty state. Rename sidebar footer label from “Store Log” to “Logs”. Notes: Displays latest error/warning lines; create or clear .log files under log_store.zarr/logs to control output. No breaking changes; existing S3 endpoints unaffected.
…ading pathModularize backend env + docs; remove legacy requirements; fix env loading path Description: Backend now loads environment only from app/databuk/dashboard/backend/.env (no root fallback) Fixed loader path in core/config.py to point to backend directory Removed hardcoded root .env loading from backend/main.py Deleted redundant backend/requirements.txt (using pyproject.toml) Updated app/databuk/dashboard/backend/README.md with new setup, endpoints, and env instructions Updated app/databuk/dashboard/README.md to reflect backend packaging, .env location, and run commands Minor docs cleanup and endpoint listings (s3, logs, config)
#39) * feat(ci,helm,oci): prepared containerfiles, helm chart and CI for dashboard * fix(ci): workflow trigger * fix(ci): removed error from custom actions * fix(ci): add registry username and password for containerize action * fix(ci): removed error msg * feat(ci): small improvements * fix(ci): changed artifact path * fix(ci): chanched concurrency * feat(ci): improved concurrency and testing deploy * fea(ci,helm): changed conteinerize for ingress server and add seccompProfile to security context for dashboard chart * fix(helm): changed configmap name * feat(ci,helm): small improvements * fix(helm): configmap * feaet(react,helm): changed base url for api calls * feat(dashboard): add env development * fix(dashboard): self review * fix(helm): removed rewrite target annotaion * fix(ci): change vite api url * fix(ci): removed deploy for dashboard pull request
- Add Use_zarr_fuse flag to endpoints.yaml and ConfigManager - Add zarr_fuse import and path resolution in S3Service - Add flag-controlled routing for get_variable_data and get_store_structure - Fix double /api prefix issue in frontend requests
- Add _get_store_structure_zarr_fuse using zarr_fuse's internal store opening - Switch to xarray.open_zarr due to FsspecStore compatibility issues - Add _convert_to_legacy_format to match frontend expectations - Fix S3 endpoint URL mapping for proper connection
| @@ -182,3 +182,25 @@ cython_debug/ | |||
|
|
|||
| # PyPI configuration file | |||
| .pypirc | |||
There was a problem hiding this comment.
General points:
- put all frontend stuff under .../dashboard/frontend
- create a docker file for full release testing and simple daplyment, something like:
# --- build frontend (builder stage) ---
FROM node:20-alpine AS web-build
WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN npm ci
COPY frontend/ .
RUN npm run build # outputs to frontend/dist
# --- backend runtime (final image) ---
FROM python:3.12-slim
WORKDIR /app
# install backend
COPY backend/pyproject.toml backend/ .
RUN pip install --no-cache-dir .
# copy static assets produced by Vite
COPY --from=web-build /app/frontend/dist /app/app/static
# run
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
There was a problem hiding this comment.
Maybe I can create docker compose and deployment OCI(Containerfile), if you need it. @jbrezmorf
There was a problem hiding this comment.
Already done in OCI.
| @@ -0,0 +1,6 @@ | |||
| venv/ | |||
There was a problem hiding this comment.
Put these into .../dashboard/.gitignore
| @@ -0,0 +1,96 @@ | |||
| # ZARR FUSE Dashboard Backend | |||
There was a problem hiding this comment.
Merge with .../dashboard/README.md
| # Base paths - Fix the path calculation | ||
| # Backend is in: app/databuk/dashboard/backend/ | ||
| # Need to go up to: zarr_fuse/ (project root) | ||
| PROJECT_ROOT = Path(__file__).parent.parent.parent.parent.parent.parent |
There was a problem hiding this comment.
Encapsulate the dashboard to its 'dashboard' folder do not use files from upper and side folders.
|
|
||
| class EndpointConfig(BaseModel): | ||
| """Pydantic model for endpoint configuration validation""" | ||
| Reload_interval: int = Field(..., gt=0, description="Reload interval in seconds") |
There was a problem hiding this comment.
- rename keys to lowercase only
- omit keys from S3_ENDPOINT_URL bellow; these should be taken from zarr_fuse.open_store logic reading from: schema, env varaibles and passed arguments
| Version: str = Field(default="1.0.0", description="Version") | ||
| Use_zarr_fuse: bool = Field(default=False, description="Enable zarr_fuse integration") | ||
|
|
||
| class ConfigManager: |
There was a problem hiding this comment.
Replace this class by pure functions, taking the path to the config -> produce dict of EndPointconfig objects. Avoiding env substitution logic which is part of zarr_fuse already it
is a single two line function.
| ] | ||
|
|
||
| # Feature flags | ||
| ENABLE_TEST_STORES: bool = True |
There was a problem hiding this comment.
Should be given by the endpoints config file (presence of test endpoints)
| CURRENT_DIR = Path(__file__).resolve().parent.parent # points to backend/ | ||
|
|
||
| # Strictly load only backend-local .env (no fallback to repo root) | ||
| BACKEND_ENV_PATH = CURRENT_DIR / ".env" |
There was a problem hiding this comment.
Remove .env injection. It leads to confusing behavior (shadowing real system variables).
|
|
||
| # S3/Zarr ekosistemi + backend web bağımlılıkları | ||
| dependencies = [ | ||
| "aiobotocore>=2.18.0", |
There was a problem hiding this comment.
List only direct dependencies.
| if __name__ == "__main__": | ||
| import uvicorn | ||
| uvicorn.run( | ||
| "main:app", |
There was a problem hiding this comment.
- import at the module level
- uvicorn.run parameter less cryptic by passing the 'app' object directly.
| async def root(): | ||
| """Root endpoint with API information.""" | ||
| return { | ||
| "message": f"Welcome to {settings.PROJECT_NAME}", |
There was a problem hiding this comment.
Keep only essential data, document the meaning of the keys for the frontend.
| "version": settings.VERSION, | ||
| "docs": "/docs", | ||
| "redoc": "/redoc", | ||
| "api_prefix": settings.API_V1_STR |
There was a problem hiding this comment.
Do not pass api prefix to the frontend. Just define it as a constant in the frontend.
The point is to have possible separation of different api versions. If the frontend uses
api prefix passed by the backend, it is like do not have api versions at all.
| @@ -0,0 +1,2 @@ | |||
| # Routers package | |||
There was a problem hiding this comment.
just keep empty, this codes does effectively nothing
| ) | ||
|
|
||
| # Include routers | ||
| app.include_router(config.router, prefix=settings.API_V1_STR) |
There was a problem hiding this comment.
I suggest to create single landing page (root endpoint) with list of endpoints = separate dasaset dashboards and then one endpoint the item in the endpoints.yaml
this structure should be reflected by the backend API as well. So you should generate first level routers according to the items in the endpoints.yaml. Then config, s3, log routers should not be module wise instances but sub routers of the endpoints.
There was a problem hiding this comment.
Moreover to manage state and possibly some cache for intividual routers we should introduce our own class per enpoint/store/separate dashboard:
routers/factory.py
from fastapi import APIRouter
from typing import Any, Dict
class ServiceAPI:
def init(self, name: str, cfg: Dict[str, Any]):
self.name = name
self.cfg = cfg
self.state = {"hits": 0}
self._router = APIRouter(prefix=f"/{name}", tags=[name])
# bind methods (self is captured)
self._router.add_api_route("/status", self.status, methods=["GET"])
self._router.add_api_route("/config", self.get_config, methods=["GET"])
self._router.add_api_route("/touch", self.touch, methods=["POST"])
@property
def router(self) -> APIRouter:
return self._router
# handlers (self available)
async def status(self):
return {"service": self.name, "hits": self.state["hits"]}
async def get_config(self):
return self.cfg
async def touch(self):
self.state["hits"] += 1
return {"ok": True, "hits": self.state["hits"]}
No description provided.