Skip to content

Mf/feature/dashboard#31

Open
jbrezmorf wants to merge 14 commits intomainfrom
MF/feature/dashboard
Open

Mf/feature/dashboard#31
jbrezmorf wants to merge 14 commits intomainfrom
MF/feature/dashboard

Conversation

@jbrezmorf
Copy link
Copy Markdown
Contributor

No description provided.

mfatihakbas and others added 14 commits August 28, 2025 10:35
- Add collapsible sidebar with smooth transitions
- Implement hierarchical tree view with folder/file icons
- Add responsive design for collapsed/expanded states
- Add custom scrollbars and improved button interactions
- Fix TailwindCSS configuration and remove unused packages
- Consolidate .gitignore files
- Add comprehensive README with setup instructions

Dashboard now has a fully functional sidebar with:
- Expandable tree structure
- Collapsible sidebar with icon-only view
- Modern gradient header design
- Responsive layout adjustments
…n logging

Major Features:
- Generic Zarr store structure support (any hierarchy)
- Real-time S3 connection with status indicators
- Accordion-style variable data viewer
- YAML-configurable endpoints
- Root/nested variable navigation
- NaN value JSON serialization
- Comprehensive error handling

Technical Improvements:
- Clean separation of S3Service and UI components
- Zarr-native path handling (removed artificial root prefix)
- Code cleanup and refactoring
- Multi-store testing capability
- Professional logging output
- Updated documentation and gitignore

Backend Changes:
- Generic S3Service with zarr_fuse integration
- Robust error handling and fallback mechanisms
- YAML-based endpoint management
- Clean log messages without visual clutter

Frontend Changes:
- Complete UI overhaul for generic store support
- Accordion-style variable display
- Real-time status indicators with progress bar
- Clickable store navigation
- Error resilient design

Configuration:
- Multiple S3 bucket support via env.example
- Flexible YAML endpoint configuration
- Clean development environment setup
Backend
Add routers/logs.py with GET /api/logs?limit= returning only error/warning entries from latest file in log_store.zarr/logs.
Add LOGS_DIR to core/config.py.
Wire router in backend/main.py.
Frontend
Update LogPanel.tsx to fetch from /api/logs, remove All/Store filters and mock data, show “Backend” badge and “No logs yet” empty state.
Rename sidebar footer label from “Store Log” to “Logs”.
Notes:
Displays latest error/warning lines; create or clear .log files under log_store.zarr/logs to control output.
No breaking changes; existing S3 endpoints unaffected.
…ading pathModularize backend env + docs; remove legacy requirements; fix env loading path

Description:
Backend now loads environment only from app/databuk/dashboard/backend/.env (no root fallback)
Fixed loader path in core/config.py to point to backend directory
Removed hardcoded root .env loading from backend/main.py
Deleted redundant backend/requirements.txt (using pyproject.toml)
Updated app/databuk/dashboard/backend/README.md with new setup, endpoints, and env instructions
Updated app/databuk/dashboard/README.md to reflect backend packaging, .env location, and run commands
Minor docs cleanup and endpoint listings (s3, logs, config)
#39)

* feat(ci,helm,oci): prepared containerfiles, helm chart and CI for dashboard

* fix(ci): workflow trigger

* fix(ci): removed error from custom actions

* fix(ci): add registry username and password for containerize action

* fix(ci): removed error msg

* feat(ci): small improvements

* fix(ci): changed artifact path

* fix(ci): chanched concurrency

* feat(ci): improved concurrency and testing deploy

* fea(ci,helm): changed conteinerize for ingress server and add seccompProfile to security context for dashboard chart

* fix(helm): changed configmap name

* feat(ci,helm): small improvements

* fix(helm): configmap

* feaet(react,helm): changed base url for api calls

* feat(dashboard): add env development

* fix(dashboard): self review

* fix(helm): removed rewrite target annotaion

* fix(ci): change vite api url

* fix(ci): removed deploy for dashboard pull request
- Add Use_zarr_fuse flag to endpoints.yaml and ConfigManager
- Add zarr_fuse import and path resolution in S3Service
- Add flag-controlled routing for get_variable_data and get_store_structure
- Fix double /api prefix issue in frontend requests
- Add _get_store_structure_zarr_fuse using zarr_fuse's internal store opening
- Switch to xarray.open_zarr due to FsspecStore compatibility issues
- Add _convert_to_legacy_format to match frontend expectations
- Fix S3 endpoint URL mapping for proper connection
Comment thread .gitignore
@@ -182,3 +182,25 @@ cython_debug/

# PyPI configuration file
.pypirc
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General points:

  • put all frontend stuff under .../dashboard/frontend
  • create a docker file for full release testing and simple daplyment, something like:
# --- build frontend (builder stage) ---
FROM node:20-alpine AS web-build
WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN npm ci
COPY frontend/ .
RUN npm run build              # outputs to frontend/dist

# --- backend runtime (final image) ---
FROM python:3.12-slim
WORKDIR /app
# install backend
COPY backend/pyproject.toml backend/ .
RUN pip install --no-cache-dir .
# copy static assets produced by Vite
COPY --from=web-build /app/frontend/dist /app/app/static
# run
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I can create docker compose and deployment OCI(Containerfile), if you need it. @jbrezmorf

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already done in OCI.

@@ -0,0 +1,6 @@
venv/
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put these into .../dashboard/.gitignore

@@ -0,0 +1,96 @@
# ZARR FUSE Dashboard Backend
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge with .../dashboard/README.md

# Base paths - Fix the path calculation
# Backend is in: app/databuk/dashboard/backend/
# Need to go up to: zarr_fuse/ (project root)
PROJECT_ROOT = Path(__file__).parent.parent.parent.parent.parent.parent
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Encapsulate the dashboard to its 'dashboard' folder do not use files from upper and side folders.


class EndpointConfig(BaseModel):
"""Pydantic model for endpoint configuration validation"""
Reload_interval: int = Field(..., gt=0, description="Reload interval in seconds")
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • rename keys to lowercase only
  • omit keys from S3_ENDPOINT_URL bellow; these should be taken from zarr_fuse.open_store logic reading from: schema, env varaibles and passed arguments

Version: str = Field(default="1.0.0", description="Version")
Use_zarr_fuse: bool = Field(default=False, description="Enable zarr_fuse integration")

class ConfigManager:
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace this class by pure functions, taking the path to the config -> produce dict of EndPointconfig objects. Avoiding env substitution logic which is part of zarr_fuse already it
is a single two line function.

]

# Feature flags
ENABLE_TEST_STORES: bool = True
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be given by the endpoints config file (presence of test endpoints)

CURRENT_DIR = Path(__file__).resolve().parent.parent # points to backend/

# Strictly load only backend-local .env (no fallback to repo root)
BACKEND_ENV_PATH = CURRENT_DIR / ".env"
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove .env injection. It leads to confusing behavior (shadowing real system variables).


# S3/Zarr ekosistemi + backend web bağımlılıkları
dependencies = [
"aiobotocore>=2.18.0",
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

List only direct dependencies.

if __name__ == "__main__":
import uvicorn
uvicorn.run(
"main:app",
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • import at the module level
  • uvicorn.run parameter less cryptic by passing the 'app' object directly.

Comment thread app/databuk/dashboard/backend/main.py
async def root():
"""Root endpoint with API information."""
return {
"message": f"Welcome to {settings.PROJECT_NAME}",
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep only essential data, document the meaning of the keys for the frontend.

"version": settings.VERSION,
"docs": "/docs",
"redoc": "/redoc",
"api_prefix": settings.API_V1_STR
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not pass api prefix to the frontend. Just define it as a constant in the frontend.
The point is to have possible separation of different api versions. If the frontend uses
api prefix passed by the backend, it is like do not have api versions at all.

@@ -0,0 +1,2 @@
# Routers package
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just keep empty, this codes does effectively nothing

)

# Include routers
app.include_router(config.router, prefix=settings.API_V1_STR)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to create single landing page (root endpoint) with list of endpoints = separate dasaset dashboards and then one endpoint the item in the endpoints.yaml
this structure should be reflected by the backend API as well. So you should generate first level routers according to the items in the endpoints.yaml. Then config, s3, log routers should not be module wise instances but sub routers of the endpoints.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moreover to manage state and possibly some cache for intividual routers we should introduce our own class per enpoint/store/separate dashboard:

routers/factory.py

from fastapi import APIRouter
from typing import Any, Dict

class ServiceAPI:
def init(self, name: str, cfg: Dict[str, Any]):
self.name = name
self.cfg = cfg
self.state = {"hits": 0}

    self._router = APIRouter(prefix=f"/{name}", tags=[name])
    # bind methods (self is captured)
    self._router.add_api_route("/status", self.status, methods=["GET"])
    self._router.add_api_route("/config", self.get_config, methods=["GET"])
    self._router.add_api_route("/touch", self.touch, methods=["POST"])

@property
def router(self) -> APIRouter:
    return self._router

# handlers (self available)
async def status(self):
    return {"service": self.name, "hits": self.state["hits"]}

async def get_config(self):
    return self.cfg

async def touch(self):
    self.state["hits"] += 1
    return {"ok": True, "hits": self.state["hits"]}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants