feat(alpasim-integration): continuation of closed loop integration

Model/data_parsing/README.md

@@ -8,5 +8,6 @@ Dataset loaders and utilities for AutoE2E training data.
 - **`nvidia_physical_ai/`** — [NVIDIA Autonomous Vehicle dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles) loader
 - **`map_rendering/`** — Map tile rendering and GPS-to-map conversions
 - **`kit_scenes/`** — [KITScenes](https://kitscenes.com/multimodal/) data utilities
+- **`alpasim_stream/`** — Real-time observation stream parser for NVIDIA AlpaSim closed-loop simulation (`PredictionInput` parity with `kit_scenes`)
 
 Each module provides dataset classes (`*Dataset`) and helper functions for loading camera frames, extracting egomotion, and handling map data.

Model/plugins/alpasim_driver/README.md

@@ -0,0 +1,152 @@
+# AutoE2E AlpaSim Driver Plugin
+
+This package provides the official **AutoE2E driver plugin** for [NVIDIA AlpaSim](https://github.com/NVlabs/alpasim), enabling real-time closed-loop evaluation and policy rollouts of the AutoE2E VLA driving model on the KitScenes 7-camera sensor topology.
+
+For full official setup, microservices architecture, and execution details, refer to the [NVIDIA AlpaSim GitHub Repository](https://github.com/NVlabs/alpasim).
+
+---
+
+## Architecture Overview
+
+The plugin connects AutoE2E directly to AlpaSim's microservices simulation loop without custom networking overhead.
+
+```mermaid
+graph TD
+    AlpaSim[AlpaSim Simulation Runtime] -->|PredictionInput: 7 RGB cams, speed, accel, command| DriverPlugin[AutoE2EDriver Plugin]
+    DriverPlugin --> Parser[AlpasimStreamParser]
+    Parser -->|Normalized Tensors| Model[AutoE2E PyTorch Model]
+    Model -->|Trajectory Waypoints + Headings| DriverPlugin
+    DriverPlugin -->|ModelPrediction: trajectory_xy, headings| AlpaSim
+```
+
+### Key Components
+
+- **`AutoE2EDriver`** ([`plugin.py`](./alpasim_autoe2e/plugin.py)): Subclass of AlpaSim's `BaseTrajectoryModel`. Implements `from_config()`, `camera_ids`, `context_length`, `output_frequency_hz`, and `predict()`.
+- **`AutoE2EAlpaSimConfig`** ([`config.py`](./alpasim_autoe2e/config.py)): Dataclass defining model checkpoint paths, dynamic camera topology configuration, and trajectory horizon parameters.
+- **Entry Points** ([`pyproject.toml`](./pyproject.toml)): Registers `autoe2e` under entry point groups `alpasim.models` and `alpasim.configs`.
+- **Driver Configs** ([`configs/driver/`](./alpasim_autoe2e/configs/driver/)): Contains `autoe2e.yaml` and `autoe2e_configs.yaml`. These files formally register the 7-camera KIT topology with AlpaSim to override the default renderer camera setup, avoiding `KeyError`s during closed-loop simulation.
+
+---
+
+## Data Contract & Sensor Topology
+
+### Input Observations (`PredictionInput`)
+- **Visual Topology**: 7 KitScenes camera streams (`camera_base_front_center`, `camera_ring_front`, `camera_ring_front_left`, `camera_ring_front_right`, `camera_ring_rear`, `camera_ring_rear_left`, `camera_ring_rear_right`).
+- **Telemetry**: Scalar ego vehicle speed (*m/s*), acceleration (*m/s²*), yaw rate (*rad/s*), and trajectory curvature (*1/m*), alongside a `route_mask` natively rendered from the scene's dynamic `ego_pose`.
+
+### Output Predictions (`ModelPrediction`)
+- **`trajectory_xy`**: Waypoint coordinates *[64, 2]* in rig frame (*X* forward, *Y* left).
+- **`headings`**: Vehicle target headings *[64]* in radians.
+
+---
+
+## Installation & Setup
+
+### 1. Install Driver & Dependencies
+
+Install the driver plugin and dataset parser in editable mode:
+
+```bash
+# 1. Install alpasim_driver plugin package
+pip install -e Model/plugins/alpasim_driver
+
+# 2. Install KITScenes SDK
+pip install -e Model/data_parsing/kit_scenes/kitscenes --no-deps
+
+# 3. Install Lanelet2 (for vector HD map parsing & BEV rasterization)
+pip install lanelet2
+```
+
+### 2. Environment Configuration
+
+Configure root directories for KITScenes dataset files and AlpaSim source repository. You can source them from `.env` or export them manually:
+
+```bash
+# Option A: Load from .env file
+set -a; source .env; set +a
+
+# Option B: Set environment variables manually
+export KITSCENES_ROOT="/path/to/your/dataset/directory"
+export ALPASIM_ROOT="/path/to/alpasim/repository"
+```
+
+### 3. Download KITScenes Data Samples
+
+Download dataset scene archives using the `kitscenes` CLI:
+
+```bash
+python -m kitscenes.download "$KITSCENES_ROOT" --scenes <scene_id> # for example c34c778f-ad8c-0aa9-7e1a-c86a73f887c7
+```
+
+---
+
+## Model Control Parameters
+
+Controls for simulation execution in [`config.py`](./alpasim_autoe2e/config.py) and [`plugin.py`](./alpasim_autoe2e/plugin.py):
+
+| Parameter | Type | Default | Description |
+| :--- | :--- | :--- | :--- |
+| `checkpoint_path` | `str` | `"autoe2e_model.ckpt"` | Path to pre-trained AutoE2E PyTorch checkpoint file. |
+| `allow_untrained_model` | `bool` | `False` | When `True`, initializes a fresh `AutoE2E` PyTorch neural network with random weights (dynamically scaled to `num_views=len(camera_ids)`) if no checkpoint file exists on disk. |
+| `allow_mock` | `bool` | `False` | When `False` (default), strictly requires the actual AlpaSim runtime and real model execution, failing fast if dependencies are missing. |
+
+---
+
+## Plugin Discovery Verification
+
+Confirm that AlpaSim discovers the `autoe2e` plugin entry points:
+
+```python
+import alpasim_autoe2e.plugin
+import alpasim_plugins.plugins as p
+
+print("Registered Models:", p.PluginRegistry("alpasim.models").get_names())
+print("Registered Configs:", p.PluginRegistry("alpasim.configs").get_names())
+```
+
+**Expected Output**:
+```text
+Registered Models: ['autoe2e']
+Registered Configs: ['autoe2e']
+```
+
+---
+
+## Workflows & Official Documentation
+
+### 1. Build the Driver Container Image
+AlpaSim automatically discovers and installs plugins located in its `plugins/` directory. Because Docker cannot resolve symlinks that point outside of its build context, you **must** hardcopy the driver plugin into `$ALPASIM_ROOT/plugins/` before building the image.
+
+From the repository root, execute:
+
+```bash
+# 1. Sync the plugin code into the AlpaSim build context
+rm -rf "$ALPASIM_ROOT/plugins/alpasim_driver"
+cp -r Model/plugins/alpasim_driver "$ALPASIM_ROOT/plugins/"
+
+# 2. Build the Docker image
+cd "$ALPASIM_ROOT"
+docker build -t alpasim-base:latest .
+cd -
+```
+
+*Note: The `AutoE2E` inference pipeline inside the simulator relies purely on `torch` and does not require the offline `kitscenes` or `lanelet2` packages, as the AlpaSim parser receives generic tensors directly.*
+
+### 2. Run the Closed-Loop Simulation
+Once the image is built, use the `alpasim_wizard` from the repository root to launch the simulation.
+
+```bash
+# From the repository root (Mock Mode for testing):
+uv run --project "$ALPASIM_ROOT/src/wizard" alpasim_wizard \
+    deploy=local \
+    topology=1gpu \
+    driver=autoe2e_mock \
+    wizard.log_dir=$PWD/outputs/autoe2e_closed_loop_run \
+    defines.base_image=alpasim-base:latest
+```
+
+*To run with a real production checkpoint, use `driver=autoe2e driver.model.checkpoint_path=/path/to/checkpoint.pt`.*
+
+*Note: For the NuRec 3DGS renderer to successfully boot and render the 7 KIT cameras, the selected dataset scene must have `.usdz` artifacts compiled and available in the scene cache.*
+
+For further details on official CLI workflows and AlpaSim architecture, refer to the [NVIDIA AlpaSim GitHub Repository](https://github.com/NVlabs/alpasim).

Model/plugins/alpasim_driver/alpasim_autoe2e/__init__.py

@@ -0,0 +1 @@
+"""AutoE2E AlpaSim Driver Plugin package."""

Model/plugins/alpasim_driver/alpasim_autoe2e/config.py

@@ -0,0 +1,109 @@
+"""Configuration dataclasses for the AutoE2E AlpaSim driver plugin.
+
+Defines model checkpoints, camera topology settings, and trajectory planning horizon settings.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+import json
+from pathlib import Path
+from typing import List, Tuple, Dict
+
+_CALIB_DIR = Path(__file__).resolve().parent / "configs" / "calibration"
+_DATASET_DIR = Path(__file__).resolve().parent / "configs" / "dataset"
+
+DEFAULT_CAMERA_NAMES: List[str] = [
+    "camera_base_front_center",
+    "camera_ring_front",
+    "camera_ring_front_left",
+    "camera_ring_front_right",
+    "camera_ring_rear",
+    "camera_ring_rear_left",
+    "camera_ring_rear_right",
+]
+
+DEFAULT_IMAGE_MEAN: List[float] = [0.485, 0.456, 0.406]
+DEFAULT_IMAGE_STD: List[float] = [0.229, 0.224, 0.225]
+
+
+def load_dataset_config(
+    dataset_config_path: str | Path | None = None,
+) -> dict:
+    """Load dataset configuration (mean, std, etc.) from JSON."""
+    path = (
+        Path(dataset_config_path) if dataset_config_path else _DATASET_DIR / "kit_scenes.json"
+    )
+    with open(path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def get_image_transform(
+    image_mean: List[float] | None = None,
+    image_std: List[float] | None = None,
+):
+    """Build image preprocessing transform using dataset-specific normalization."""
+    from torchvision import transforms
+
+    if image_mean is None or image_std is None:
+        cfg = load_dataset_config()
+        image_mean = image_mean or cfg["image_mean"]
+        image_std = image_std or cfg["image_std"]
+
+    return transforms.Compose(
+        [
+            transforms.ToTensor(),
+            transforms.Normalize(mean=image_mean, std=image_std),
+        ]
+    )
+
+
+def load_projection_matrices(
+    calibration_path: str | Path | None = None,
+) -> Dict[str, list[list[float]]]:
+    """Load camera projection matrices from a JSON calibration file."""
+    path = (
+        Path(calibration_path) if calibration_path else _CALIB_DIR / "kit_scenes.json"
+    )
+    with open(path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+@dataclass
+class AutoE2EAlpaSimConfig:
+    """Configuration options for ``AutoE2EAlpaSimModel`` driver plugin.
+
+    Registered with AlpaSim under entry point ``alpasim.configs``.
+    """
+
+    checkpoint_path: str
+    """Path to trained AutoE2E model checkpoint file."""
+
+    allow_mock: bool = False
+    """Whether to allow mock fallback mode when running without AlpaSim."""
+    allow_untrained_model: bool = False
+    """Whether to initialize the model randomly if weights are missing (useful for dry runs)."""
+
+    rewards: Dict[str, float] = field(default_factory=dict)
+    """Dictionary mapping reward component names to their scalar weights."""
+
+    image_size: Tuple[int, int] = (256, 256)
+    """Target camera resolution ``(H, W)`` expected by perception backbone."""
+
+    planning_horizon_s: float = 6.4
+    """Total future trajectory planning horizon in seconds."""
+
+    planning_steps: int = 64
+    """Number of output waypoint steps along the planning horizon."""
+
+    camera_names: List[str] = field(default_factory=lambda: list(DEFAULT_CAMERA_NAMES))
+    """List of 7 camera names matching KitScenes topology."""
+
+    scene_id: str | None = None
+    """KITScenes scene ID (e.g., 'c34c778f-...') to load offline map and trajectory masks natively."""
+
+    image_mean: List[float] = field(default_factory=lambda: list(DEFAULT_IMAGE_MEAN))
+    """Mean per RGB channel for input image normalization."""
+
+    image_std: List[float] = field(default_factory=lambda: list(DEFAULT_IMAGE_STD))
+    """Standard deviation per RGB channel for input image normalization."""

Model/plugins/alpasim_driver/alpasim_autoe2e/configs/calibration/kit_scenes.json

@@ -0,0 +1,37 @@
+{
+  "camera_base_front_center": [
+    [128.755274, -131.199077, 1.006187, -52.390168],
+    [127.524094, -0.908146, -249.183598, -149.383547],
+    [0.999974, 0.007067, 0.000368, -0.42134]
+  ],
+  "camera_ring_front": [
+    [129.386719, -132.323571, 0.863261, -25.981988],
+    [128.731816, 1.228626, -207.188816, -64.53605],
+    [0.999854, 0.016975, 0.00191, -0.207052]
+  ],
+  "camera_ring_front_left": [
+    [180.400433, 47.059563, -0.395239, -26.819215],
+    [62.00389, 109.554406, -208.719258, -64.554699],
+    [0.48647, 0.873665, -0.007472, -0.209015]
+  ],
+  "camera_ring_front_right": [
+    [-48.124751, -179.391707, 0.367795, -26.278596],
+    [66.340567, -109.558404, -207.977877, -64.484149],
+    [0.52269, -0.852522, -0.000715, -0.206905]
+  ],
+  "camera_ring_rear": [
+    [-131.283609, 132.521569, -0.505595, -26.582819],
+    [-128.653595, -3.802507, -207.379122, -64.645621],
+    [-0.99984, -0.017437, 0.003915, -0.203476]
+  ],
+  "camera_ring_rear_left": [
+    [48.199949, 180.3027, -1.179315, -26.830027],
+    [-65.506368, 107.897468, -208.745168, -64.714219],
+    [-0.518128, 0.85526, -0.008521, -0.205967]
+  ],
+  "camera_ring_rear_right": [
+    [-179.652773, -48.899689, 1.256024, -25.914482],
+    [-63.016182, -113.192575, -207.988609, -64.629469],
+    [-0.47451, -0.880249, 0.000901, -0.203962]
+  ]
+}

Model/plugins/alpasim_driver/alpasim_autoe2e/configs/dataset/kit_scenes.json

@@ -0,0 +1,5 @@
+{
+  "dataset_name": "kit_scenes",
+  "image_mean": [0.485, 0.456, 0.406],
+  "image_std": [0.229, 0.224, 0.225]
+}