diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3df8df57b..e0e2eff25 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,258 +1,72 @@ # Contributing -Thank you for your interest in contributing to Jumpstarter! This document outlines the process for contributing to the project and provides guidelines to make the contribution process smooth. +Thank you for your interest in contributing to Jumpstarter! ## Getting Help -If you have questions or need help, here are the best ways to connect with the community: - -### Community Resources - -- **Matrix Chat**: Join our [Matrix community](https://matrix.to/#/#jumpstarter:matrix.org) for real-time discussions and support -- **Weekly Meetings**: Participate in our [weekly community meetings](https://meet.google.com/gzd-hhbd-hpu) to discuss development and get help -- **Etherpad**: Check our [Etherpad](https://etherpad.jumpstarter.dev/pad-lister) for meeting notes and collaborative documentation -- **GitHub Issues**: [Open an issue](https://github.com/jumpstarter-dev/jumpstarter/issues) in the repository for bug reports and feature requests -- **Documentation**: Visit our [documentation](https://jumpstarter.dev/) for comprehensive guides and tutorials - -## Code of Conduct - -Please be respectful and considerate of others when contributing to the project. +- **Matrix Chat**: Join our [Matrix community](https://matrix.to/#/#jumpstarter:matrix.org) +- **GitHub Issues**: [Open an issue](https://github.com/jumpstarter-dev/jumpstarter/issues) +- **Documentation**: Visit our [documentation](https://jumpstarter.dev/) +- **Weekly Meetings**: [Google Meet](https://meet.google.com/gzd-hhbd-hpu) +- **Etherpad**: [Collaborative docs](https://etherpad.jumpstarter.dev/pad-lister) ## Getting Started 1. Fork the repository 2. Clone your fork locally -3. Set up the development environment -4. Make your changes on a new branch +3. Set up the development environment: `make sync` and `make test` +4. Make changes on a new branch 5. Test your changes thoroughly 6. Submit a pull request -## Development Setup - -```shell -# Install dependencies -make sync - -# Run tests -make test -``` - ## Contribution Guidelines ### Making Changes -- Keep changes focused and related to a single issue -- Follow the existing code style and conventions -- Add tests for new features or bug fixes -- Update documentation as needed +- Focus on a single issue +- Follow existing code style +- Add tests and update documentation ### Commit Messages -- Use clear and descriptive commit messages +- Use clear, descriptive messages - Reference issue numbers when applicable - Follow conventional commit format when possible ### Pull Requests -- Provide a clear description of the changes -- Link to any relevant issues -- Ensure all tests pass before submitting -- Be responsive to feedback and questions +- Provide a clear description +- Link to relevant issues +- Ensure all tests pass ## Types of Contributions ### Code Contributions - -We welcome contributions to the core codebase, including bug fixes, features, and improvements. +We welcome bug fixes, features, and improvements to the core codebase. ### Contributing Drivers -If you are working on a driver or adapter library of general interest, please consider submitting it to this repository, as it will become available to all Jumpstarter users. - -To create a new driver scaffold, you can use the `create_driver.sh` script: +To create a new driver scaffold: ```shell -./__templates__/create_driver.sh vendor_name driver_name "Your Name" "your.email@example.com" -``` - -This will create the necessary files and structure for your driver in the `packages/` directory. For example: - -```shell -./__templates__/create_driver.sh yepkit Ykush "Your Name" "your.email@example.com" -``` - -Creates: -``` -packages/jumpstarter_driver_yepkit/jumpstarter_driver_yepkit/__init__.py -packages/jumpstarter_driver_yepkit/jumpstarter_driver_yepkit/client.py -packages/jumpstarter_driver_yepkit/jumpstarter_driver_yepkit/driver_test.py -packages/jumpstarter_driver_yepkit/jumpstarter_driver_yepkit/driver.py -packages/jumpstarter_driver_yepkit/.gitignore -packages/jumpstarter_driver_yepkit/pyproject.toml -packages/jumpstarter_driver_yepkit/README.md -packages/jumpstarter_driver_yepkit/examples/exporter.yaml +$ ./__templates__/create_driver.sh driver_package DriverClass "Your Name" "your.email@example.com" ``` -#### Driver Structure - -A Jumpstarter driver typically consists of: - -1. **Driver Implementation**: The core functionality that interfaces with the device or service -2. **Client Implementation**: Client code to interact with the driver -3. **Tests**: Tests to verify the driver's functionality -4. **Examples**: Example configurations showing how to use the driver -5. **Documentation**: Clear documentation on setup and usage - -#### Private Drivers - -If you are creating a driver or adapter library for a specific need, not of general interest, or that needs to be private, please consider forking our [jumpstarter-driver-template](https://github.com/jumpstarter-dev/jumpstarter-driver-template). - -#### Driver Development Workflow - -After creating your driver skeleton: - -1. Implement the driver functionality according to the Jumpstarter driver interface -2. Write comprehensive tests for your driver -3. Create example configurations -4. Document the setup and usage in the README.md -5. Submit a pull request to the main Jumpstarter repository - -#### Testing Your Driver - -To test your driver during development: - -```shell -# From the project root -make sync # Synchronize dependencies -cd packages/your_driver_package -pytest # Run tests for your driver -``` - -#### Driver Best Practices - -- Follow the existing code style and conventions -- Write comprehensive documentation -- Include thorough test coverage -- Create example configurations for common use cases -- Keep dependencies minimal and well-justified - -### Documentation Contributions - -We welcome improvements to our documentation. - -#### Documentation System Overview +For private drivers, consider forking our [jumpstarter-driver-template](https://github.com/jumpstarter-dev/jumpstarter-driver-template). -Jumpstarter uses [Sphinx](https://www.sphinx-doc.org/en/master/) with Markdown support for its documentation. The documentation is organized into various sections covering different aspects of the project. +Test your driver: `make pkg-test-${package_name}` -#### Setting Up Documentation Environment +### Contributing Documentation -To contribute to the documentation, you'll need to set up your development environment: - -1. Clone the Jumpstarter repository -2. Navigate to the docs directory -3. Install dependencies (if not already installed with the main project) - -#### Building the Documentation Locally - -To build and preview the documentation locally: +Jumpstarter uses Sphinx with Markdown. Build and preview locally: ```shell -cd docs -make html # Build HTML documentation -make docs-serve # Serve documentation locally for preview -``` - -The documentation will be available at http://localhost:8000 in your web browser. - -#### Documentation Structure - -- `docs/source/`: Root directory for documentation source files - - `index.md`: Main landing page - - `*.md`: Various markdown files for documentation sections - - `_static/`: Static assets like images and CSS - - `_templates/`: Custom templates - -#### Writing Documentation - -Documentation is written in Markdown with some Sphinx-specific extensions. Common syntax includes: - -```markdown -# Heading 1 -## Heading 2 -### Heading 3 - -*italic text* -**bold text** - -- Bullet list item -- Another item - -1. Numbered list -2. Second item - -[Link text](URL) - -![Image alt text](path/to/image.png) - -```{note} -This is a note admonition -``` - -```{warning} -This is a warning admonition +$ make docs-serve ``` -```python -# This is a code block -def example(): - print("Hello, world!") -``` -``` - -#### Documentation Style Guide - -Please follow these guidelines when writing documentation: - -1. Use clear, concise language -2. Write in the present tense -3. Use active voice when possible -4. Include practical examples -5. Break up text with headers, lists, and code blocks -6. Target both beginners and advanced users with appropriate sections -7. Provide cross-references to related documentation - -#### Adding New Documentation Pages - -To add a new documentation page: - -1. Create a new Markdown (`.md`) file in the appropriate directory -2. Add the page to the relevant `index.md` or `toctree` directive -3. Build the documentation to ensure it appears correctly - -#### Documentation Versioning - -Documentation is versioned alongside the main Jumpstarter releases. When making changes, consider whether they apply to the current version or future versions. - -#### Documentation Tips - -- **Screenshots**: Include screenshots for complex UI operations -- **Command Examples**: Always include example commands with expected output -- **Troubleshooting**: Add common issues and their solutions -- **Links**: Link to relevant external resources when appropriate - -#### Submitting Documentation Changes - -Once your documentation changes are complete: - -1. Build the documentation to verify there are no errors -2. Submit a pull request with your changes -3. Respond to feedback during the review process - -### Example Contributions - -To add a new example: +Documentation recommended practices: -1. Create a new directory in the `examples/` folder with a descriptive name -2. Include a comprehensive `README.md` with setup and usage instructions -3. Follow the structure of existing examples -4. Ensure the example is well-documented and easy to follow +- Use clear, concise language +- Include practical examples +- Break up text with headers, lists, and code blocks +- Target both beginners and advanced users \ No newline at end of file diff --git a/README.md b/README.md index 6bde3b46a..ff7fbcbeb 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# ![bolt](./assets/bolt.svg) Jumpstarter +# ![bolt](assets/bolt.svg) Jumpstarter [![Matrix](https://img.shields.io/matrix/jumpstarter%3Amatrix.org?color=blue)](https://matrix.to/#/#jumpstarter:matrix.org) [![Etherpad](https://img.shields.io/badge/Etherpad-Notes-blue?logo=etherpad)](https://etherpad.jumpstarter.dev/pad-lister) @@ -8,13 +8,12 @@ A free, open source tool for automated testing on real and virtual hardware with ## Highlights -- ๐Ÿš€ **Unified Testing** - One tool for local, virtual, and remote hardware +- ๐Ÿงช **Unified Testing** - One tool for local, virtual, and remote hardware - ๐Ÿ **Python-Powered** - Leverage Python's testing ecosystem -- ๐Ÿ› ๏ธ **Hardware Abstraction** - Simplify complex hardware interfaces with drivers -- ๐Ÿ”Œ Built-in support for common interfaces such as [CAN](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-driver-can), [IP](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-driver-network), [GPIO](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-driver-raspberrypi), [U-Boot](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-driver-uboot), [SD Wire](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-driver-sdwire), etc. -- ๐Ÿ’ป **Collaborative** - Share test hardware globally -- ๐Ÿ”„ **CI/CD Ready** - Works with cloud native developer environments and pipelines -- ๐Ÿ–ฅ๏ธ **Cross-Platform** - Supports Linux and macOS +- ๐Ÿ”Œ **Hardware Abstraction** - Simplify complex hardware interfaces with drivers +- ๐ŸŒ **Collaborative** - Share test hardware globally +- โš™๏ธ **CI/CD Ready** - Works with cloud native developer environments and pipelines +- ๐Ÿ’ป **Cross-Platform** - Supports Linux and macOS ## Installation diff --git a/__templates__/create_driver.sh b/__templates__/create_driver.sh index abd3bc9dc..f80ca2754 100755 --- a/__templates__/create_driver.sh +++ b/__templates__/create_driver.sh @@ -28,7 +28,7 @@ mkdir -p ${MODULE_DIRECTORY} mkdir -p ${DRIVER_DIRECTORY}/examples # Define paths -DOCS_DIRECTORY=docs/source/api-reference/drivers +DOCS_DIRECTORY=docs/source/reference/package-apis/drivers DOC_FILE=${DOCS_DIRECTORY}/${DRIVER_NAME}.md README_FILE=${DRIVER_DIRECTORY}/README.md diff --git a/docs/source/api-reference/adapters.md b/docs/source/api-reference/adapters.md deleted file mode 100644 index b13e96fc6..000000000 --- a/docs/source/api-reference/adapters.md +++ /dev/null @@ -1,129 +0,0 @@ -# Adapter Classes and Architecture - -## Adapter Architecture - -Jumpstarter adapters transform connections established by drivers into different forms or interfaces. The architecture follows a pattern with these key components: - -- **Adapter Base** - Adapters typically follow a context manager pattern using Python's `with` statement for resource management. Each adapter takes a driver client as input and transforms its connection. - -- **Connection Transformation** - Adapters create a new interface on top of an existing driver connection, such as forwarding ports, providing web interfaces, or offering terminal-like access. - -- **Resource Lifecycle** - Adapters handle proper setup and teardown of resources, ensuring connections are properly established and cleaned up. - -Unlike drivers, which have a strict client/server architecture, adapters operate entirely on the client side and transform existing connections rather than establishing new ones directly with hardware or virtual devices. - -## Adapter Types - -Different types of adapters serve different needs: - -- **Port Forwarding Adapters** - Convert network connections to local ports or sockets -- **Interactive Adapters** - Provide interactive shells or console-like interfaces -- **Protocol Adapters** - Transform connections to use different protocols (e.g., SSH, VNC) -- **UI Adapters** - Create user interfaces for interacting with devices (e.g., web-based VNC) - -## Implementation Patterns - -Adapters typically implement the context manager protocol (`__enter__` and `__exit__`) to ensure proper resource management. The general pattern is: - -1. Initialize with a driver client reference -2. Set up the transformed connection in `__enter__` -3. Return the appropriate interface (URL, address, interactive object) -4. Clean up resources in `__exit__` - -This allows adapters to be used in `with` statements for clean, deterministic resource handling. - -## Example Implementation - -```{testcode} -from contextlib import contextmanager -import socket -import threading -from typing import Tuple, Any - -class TcpPortforwardAdapter: - """ - Adapter that forwards a remote TCP port to a local TCP port. - - Args: - client: A network driver client that provides a connection - local_host: Host to bind to (default: 127.0.0.1) - local_port: Port to bind to (default: 0, which selects a random port) - - Returns: - A tuple of (host, port) when used as a context manager - """ - def __init__(self, client, local_host="127.0.0.1", local_port=0): - self.client = client - self.local_host = local_host - self.local_port = local_port - self._server = None - self._thread = None - - def __enter__(self) -> Tuple[str, int]: - # Create a socket server - self._server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self._server.bind((self.local_host, self.local_port)) - self._server.listen(5) - - # Get the actual port (if we used port 0) - self.local_host, self.local_port = self._server.getsockname() - - # Start a thread to handle connections - self._thread = threading.Thread(target=self._handle_connections, daemon=True) - self._thread.start() - - return (self.local_host, self.local_port) - - def __exit__(self, exc_type, exc_val, exc_tb): - if self._server: - self._server.close() - self._server = None - - # Thread will exit because it's a daemon - self._thread = None - - def _handle_connections(self): - while True: - try: - client_socket, _ = self._server.accept() - # For each connection, establish a connection to the remote - # and set up bidirectional forwarding - remote_conn = self.client.connect() - self._start_forwarding(client_socket, remote_conn) - except Exception: - # Server was closed or other error - break - - def _start_forwarding(self, local_socket, remote_conn): - # Set up bidirectional forwarding between local_socket and remote_conn - # Typically done with two threads, one for each direction - # Implementation details depend on the specific driver client interface - pass - - -# Example usage: -def example_usage(): - # Assuming 'client' is a network driver client - with TcpPortforwardAdapter(client, local_port=8080) as (host, port): - print(f"Service available at {host}:{port}") - # The service is now accessible at the local address - # while this context is active -``` - -## Advanced Usage - -Adapters can be composed and extended for more complex scenarios: - -- **Chaining adapters**: Use the output of one adapter as the input to another -- **Custom adapters**: Create specialized adapters for specific hardware or software interfaces -- **Extended functionality**: Add logging, monitoring, or security features on top of base adapters - -## Best Practices - -When working with adapters: - -1. Always use context managers (`with` statements) to ensure proper resource cleanup -2. Consider security implications when forwarding ports or providing network access -3. Implement proper error handling and retries for robust connections -4. Use appropriate timeouts to prevent hanging connections -5. Consider performance implications for long-running connections or high-throughput scenarios diff --git a/docs/source/api-reference/adapters/index.md b/docs/source/api-reference/adapters/index.md deleted file mode 100644 index ff8460f88..000000000 --- a/docs/source/api-reference/adapters/index.md +++ /dev/null @@ -1,13 +0,0 @@ -# Adapter Packages - -This section provides documentation on Jumpstarter adapter packages that help interface with different network protocols and systems. Adapters serve as intermediate layers for connecting Jumpstarter with various external components. - -## Types of Adapters - -* **[Network](network.md)** - Network protocol adapters and utilities - -```{toctree} -:maxdepth: 1 -:hidden: -network.md -``` diff --git a/docs/source/api-reference/adapters/network.md b/docs/source/api-reference/adapters/network.md deleted file mode 100644 index c79a3a626..000000000 --- a/docs/source/api-reference/adapters/network.md +++ /dev/null @@ -1,108 +0,0 @@ -# Network adapters - -Network adapters are for transforming network connections exposed by drivers - -```{eval-rst} -.. autofunction:: jumpstarter_driver_network.adapters.TcpPortforwardAdapter -``` - -```{eval-rst} -.. autofunction:: jumpstarter_driver_network.adapters.UnixPortforwardAdapter -``` - -```{eval-rst} -.. autofunction:: jumpstarter_driver_network.adapters.NovncAdapter -``` - -```{eval-rst} -.. autofunction:: jumpstarter_driver_network.adapters.PexpectAdapter -``` - -```{eval-rst} -.. autofunction:: jumpstarter_driver_network.adapters.FabricAdapter -``` - -## Examples -```yaml -export: - tcp_port: - type: "jumpstarter_driver_network.driver.TcpNetwork" - config: - host: localhost - port: 80 - unix_socket: - type: "jumpstarter_driver_network.driver.UnixNetwork" - config: - path: /tmp/test.sock -``` - -### Forward a remote TCP port to a local TCP port - -```{doctest} ->>> # random port on localhost ->>> with TcpPortforwardAdapter(client=client.tcp_port) as addr: -... print(addr[0], addr[1]) -127.0.0.1 ... ->>> ->>> # specific address and port ->>> with TcpPortforwardAdapter(client=client.tcp_port, local_host="127.0.0.2", local_port=8080) as addr: -... print(addr[0], addr[1]) -127.0.0.2 8080 -``` - -### Forward a remote Unix domain socket to a local socket - -```{doctest} ->>> with UnixPortforwardAdapter(client=client.unix_socket) as addr: -... print(addr) -/.../socket ->>> # the type of the remote socket and the local one doesn't have to match ->>> # e.g. forward a remote Unix domain socket to a local TCP port ->>> with TcpPortforwardAdapter(client=client.unix_socket) as addr: -... print(addr[0], addr[1]) -127.0.0.1 ... -``` - -Connect to a remote TCP port with a web-based VNC client - -```{doctest} ->>> with NovncAdapter(client=client.tcp_port) as url: -... print(url) # open the url in browser to access the VNC client -https://novnc.com/noVNC/vnc.html?autoconnect=1&reconnect=1&host=127.0.0.1&port=... -``` - -Interact with a remote TCP port as if it's a serial console - -See [pexpect](https://pexpect.readthedocs.io/en/stable/api/fdpexpect.html) for API documentation - -```{doctest} ->>> # the server echos all inputs ->>> with PexpectAdapter(client=client.tcp_port) as expect: -... assert expect.send("hello") == 5 # written 5 bytes -... assert expect.expect(["hi", "hello"]) == 1 # found string at index 1 -``` - -Connect to a remote TCP port with the fabric SSH client - -See [fabric](https://docs.fabfile.org/en/latest/api/connection.html#fabric.connection.Connection) for API documentation - -```{testcode} -:skipif: True -with FabricAdapter(client=client.tcp_port, connect_kwargs={"password": "secret"}) as conn: - conn.run("uname") -``` - -```{testsetup} * -from jumpstarter_driver_network.adapters import * -from jumpstarter_driver_network.driver import * -from jumpstarter_driver_composite.driver import Composite -from jumpstarter.common.utils import serve - -instance = serve(Composite(children={"tcp_port": EchoNetwork(), "unix_socket": EchoNetwork()})) - -client = instance.__enter__() -``` - -```{testcleanup} * -instance.__exit__(None, None, None) -``` diff --git a/docs/source/api-reference/drivers.md b/docs/source/api-reference/drivers.md deleted file mode 100644 index aac480527..000000000 --- a/docs/source/api-reference/drivers.md +++ /dev/null @@ -1,136 +0,0 @@ -# Driver Classes and Architecture - -## Driver Architecture - -Jumpstarter uses a client/server model for drivers to interact with hardware. -The architecture follows a pattern with these key components: - -- **Interface Class** - An abstract base class using Python's ABCMeta to define - the contract (methods and their signatures) that driver implementations must - fulfill. The interface also specifies the client class through the `client()` - class method. - -- **Driver Class** - Inherits from both the Interface and the base `Driver` - class, implementing the logic to configure and use hardware interfaces. Driver - methods are marked with the `@export` decorator to expose them over the - network. - -- **Driver Client** - Provides a user-friendly interface that can be used by - clients to interact with the driver either locally or remotely over the - network. - -When a client requests a lease and connects to an exporter, a session is created -for all tests the client needs to execute. Within this session, the specified -`Driver` subclass is instantiated for each configured interface. These driver -instances live throughout the session's duration, maintaining state and -executing setup/teardown logic. - -On the client side, a `DriverClient` subclass is instantiated for each exported -interface. Since clients may run on different machines than exporters, -`DriverClient` classes are loaded dynamically when specified in the allowed -packages list. - -To maintain compatibility, avoid making breaking changes to interfaces. Add new -methods when needed but preserve existing signatures. If breaking changes are -required, create new interface, client, and driver versions within the same -module. - -## Driver -```{eval-rst} -.. autoclass:: jumpstarter.driver.Driver - :members: -.. autodecorator:: jumpstarter.driver.export -.. autodecorator:: jumpstarter.driver.exportstream -``` - -## Driver Client -```{eval-rst} -.. autoclass:: jumpstarter.client.DriverClient - :members: -``` - -## Example Implementation -```{testcode} -from sys import modules -from types import SimpleNamespace -from anyio import connect_tcp, sleep -from contextlib import asynccontextmanager -from collections.abc import Generator, AsyncGenerator -from abc import ABCMeta, abstractmethod -from jumpstarter.driver import Driver, export, exportstream -from jumpstarter.client import DriverClient -from jumpstarter.common.utils import serve - -# Define an interface with ABCMeta -class GenericInterface(metaclass=ABCMeta): - @classmethod - def client(cls) -> str: - return "example.GenericClient" - - @abstractmethod - def query(self, param: str) -> str: ... - - @abstractmethod - def get_data(self) -> Generator[dict, None, None]: ... - - @abstractmethod - def create_stream(self): ... - -# Implement the interface with the Driver base class -class GenericDriver(GenericInterface, Driver): - @export - def query(self, param: str) -> str: - # This could be any device-specific command - return f"Response for {param}" - - # driver calls can be either sync or async - @export - async def async_query(self, param: str) -> str: - # Example of an async operation with delay - await sleep(1) - return f"Async response for {param}" - - @export - def get_data(self) -> Generator[dict, None, None]: - # Example of a streaming response - could be sensor data, logs, etc. - for i in range(3): - yield {"type": "data", "value": i, "timestamp": f"2023-04-0{i+1}"} - - # stream constructor has to be an AsyncContextManager - # that yield an anyio.abc.ObjectStream - @exportstream - @asynccontextmanager - async def create_stream(self): - # This could be any stream connection to a device - async with await connect_tcp(remote_host="example.com", remote_port=80) as stream: - yield stream - -class GenericClient(DriverClient): - # client methods are sync - def query(self, param: str) -> str: - return self.call("query", param) - - def async_query(self, param: str) -> str: - # async driver methods can be invoked the same way - return self.call("async_query", param) - - def get_data(self) -> Generator[dict, None, None]: - yield from self.streamingcall("get_data") - - # Streams can be used for bidirectional communication - def with_stream(self, callback): - with self.stream("create_stream") as stream: - callback(stream) - -modules["example"] = SimpleNamespace(GenericClient=GenericClient) - -with serve(GenericDriver()) as client: - print(client.query("test")) - assert client.async_query("async test") == "Async response for async test" - data = list(client.get_data()) - assert len(data) == 3 -``` - -```{testoutput} -Response for test -``` \ No newline at end of file diff --git a/docs/source/api-reference/drivers/can.md b/docs/source/api-reference/drivers/can.md deleted file mode 120000 index a30c88701..000000000 --- a/docs/source/api-reference/drivers/can.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-can/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/corellium.md b/docs/source/api-reference/drivers/corellium.md deleted file mode 120000 index f9d1f64cd..000000000 --- a/docs/source/api-reference/drivers/corellium.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-corellium/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/dbus.md b/docs/source/api-reference/drivers/dbus.md deleted file mode 100644 index 9348197c8..000000000 --- a/docs/source/api-reference/drivers/dbus.md +++ /dev/null @@ -1,60 +0,0 @@ -# DBus driver - -`jumpstarter-driver-dbus` provides functionality for transparently accessing the dbus on the remote machine. - -## Installation - -```shell -pip install jumpstarter-driver-dbus -``` - -## Configuration - -Example configuration: - -```{literalinclude} dbus.yaml -:language: yaml -``` - -```{doctest} -:hide: ->>> from jumpstarter.config.exporter import ExporterConfigV1Alpha1DriverInstance ->>> ExporterConfigV1Alpha1DriverInstance.from_path("source/api-reference/drivers/dbus.yaml").instantiate() -DbusNetwork(...) -``` - -## API Reference - -```{eval-rst} -.. autoclass:: jumpstarter_driver_network.client.DbusNetworkClient() - :members: -``` - -Get machine id of the remote machine - -```{doctest} ->>> with dbus: -... print(subprocess.run([ -... "busctl", -... "call", -... "org.freedesktop.systemd1", -... "/org/freedesktop/systemd1", -... "org.freedesktop.DBus.Peer", -... "GetMachineId" -... ], stdout=subprocess.PIPE).stdout.decode()) # s "34df62c767c846d5a93eb2d6f05d9e1d" -s ... -``` - -```{testsetup} * -from jumpstarter_driver_network.driver import DbusNetwork -from jumpstarter.common.utils import serve -import subprocess - -instance = serve(DbusNetwork(kind="session")) - -dbus = instance.__enter__() -``` - -```{testcleanup} * -instance.__exit__(None, None, None) -``` diff --git a/docs/source/api-reference/drivers/dutlink.md b/docs/source/api-reference/drivers/dutlink.md deleted file mode 120000 index 1bd2fa127..000000000 --- a/docs/source/api-reference/drivers/dutlink.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-dutlink/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/flashers.md b/docs/source/api-reference/drivers/flashers.md deleted file mode 120000 index fb286a1a9..000000000 --- a/docs/source/api-reference/drivers/flashers.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-flashers/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/http.md b/docs/source/api-reference/drivers/http.md deleted file mode 120000 index 41d9e15d8..000000000 --- a/docs/source/api-reference/drivers/http.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-http/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/network.md b/docs/source/api-reference/drivers/network.md deleted file mode 120000 index 5289129ef..000000000 --- a/docs/source/api-reference/drivers/network.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-network/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/opendal.md b/docs/source/api-reference/drivers/opendal.md deleted file mode 120000 index 18cc8250f..000000000 --- a/docs/source/api-reference/drivers/opendal.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-opendal/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/power.md b/docs/source/api-reference/drivers/power.md deleted file mode 120000 index 68fc38a36..000000000 --- a/docs/source/api-reference/drivers/power.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-power/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/probe-rs.md b/docs/source/api-reference/drivers/probe-rs.md deleted file mode 120000 index b16d5cb8c..000000000 --- a/docs/source/api-reference/drivers/probe-rs.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-probe-rs/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/proxy.md b/docs/source/api-reference/drivers/proxy.md deleted file mode 100644 index 610ab2fba..000000000 --- a/docs/source/api-reference/drivers/proxy.md +++ /dev/null @@ -1,39 +0,0 @@ -# Proxy driver - -`jumpstarter-driver-proxy` provides functionality for creating "proxies" or aliases of other drivers to present a desired view of the tree of devices to the client. It's also useful when several drivers need configured access to a common resource, like a serial port or a network connection. - -## Installation - -```shell -pip install jumpstarter-driver-proxy -``` - -## Configuration - -Example configuration: - -```{literalinclude} proxy.yaml -:language: yaml -``` - -## API Reference - -```{doctest} ->>> root.foo.bar.power.on() # instead of ->>> root.proxy.on() # you can do -``` - -```{testsetup} * -from jumpstarter.config.exporter import ExporterConfigV1Alpha1DriverInstance -from jumpstarter.common.utils import serve - -instance = serve( - ExporterConfigV1Alpha1DriverInstance.from_path("source/api-reference/drivers/proxy.yaml" -).instantiate()) - -root = instance.__enter__() -``` - -```{testcleanup} * -instance.__exit__(None, None, None) -``` diff --git a/docs/source/api-reference/drivers/pyserial.md b/docs/source/api-reference/drivers/pyserial.md deleted file mode 120000 index 5d36bff85..000000000 --- a/docs/source/api-reference/drivers/pyserial.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-pyserial/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/qemu.md b/docs/source/api-reference/drivers/qemu.md deleted file mode 120000 index 857cc7465..000000000 --- a/docs/source/api-reference/drivers/qemu.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-qemu/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/raspberrypi.md b/docs/source/api-reference/drivers/raspberrypi.md deleted file mode 120000 index 407825680..000000000 --- a/docs/source/api-reference/drivers/raspberrypi.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-raspberrypi/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/sdwire.md b/docs/source/api-reference/drivers/sdwire.md deleted file mode 120000 index b28d5d1ed..000000000 --- a/docs/source/api-reference/drivers/sdwire.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-sdwire/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/shell.md b/docs/source/api-reference/drivers/shell.md deleted file mode 120000 index 5bb1ea648..000000000 --- a/docs/source/api-reference/drivers/shell.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-shell/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/snmp.md b/docs/source/api-reference/drivers/snmp.md deleted file mode 120000 index f794c7298..000000000 --- a/docs/source/api-reference/drivers/snmp.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-snmp/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/tftp.md b/docs/source/api-reference/drivers/tftp.md deleted file mode 120000 index 2677379ea..000000000 --- a/docs/source/api-reference/drivers/tftp.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-tftp/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/uboot.md b/docs/source/api-reference/drivers/uboot.md deleted file mode 120000 index 4f03c7ce7..000000000 --- a/docs/source/api-reference/drivers/uboot.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-uboot/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/ustreamer.md b/docs/source/api-reference/drivers/ustreamer.md deleted file mode 120000 index 79751a6e2..000000000 --- a/docs/source/api-reference/drivers/ustreamer.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-ustreamer/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/yepkit.md b/docs/source/api-reference/drivers/yepkit.md deleted file mode 120000 index 0741324ab..000000000 --- a/docs/source/api-reference/drivers/yepkit.md +++ /dev/null @@ -1 +0,0 @@ -../../../../packages/jumpstarter-driver-yepkit/README.md \ No newline at end of file diff --git a/docs/source/api-reference/index.md b/docs/source/api-reference/index.md deleted file mode 100644 index 40a9263df..000000000 --- a/docs/source/api-reference/index.md +++ /dev/null @@ -1,18 +0,0 @@ -# API Reference - -This section provides details on the Jumpstarter core API and contrib drivers. Here you'll find comprehensive documentation for developers who want to extend or integrate with Jumpstarter. - -The following pages will guide you through the API: - -* **[Drivers](drivers/index.md)** - Documentation for infrastructure drivers -* **[Adapters](adapters/index.md)** - Documentation for adapters -* **[Driver API](drivers.md)** - Core driver API reference - -```{toctree} -:maxdepth: 1 -:hidden: -drivers/index.md -adapters/index.md -drivers.md -adapters.md -``` diff --git a/docs/source/architecture.md b/docs/source/architecture.md deleted file mode 100644 index 2a6ae7c4e..000000000 --- a/docs/source/architecture.md +++ /dev/null @@ -1,222 +0,0 @@ -# Architecture - -The Jumpstarter architecture is based on a client/server model. This enables a -single client to communicate with one or many devices under test. - -Devices can either be connected to the same machine as the client or distributed -across remote test runners, for example, in a hybrid cloud CI environment. - -The core of this architecture is the gRPC protocol that connects clients to -devices, either directly in local mode, or through a central controller in -distributed mode. - -## Core Components - -Jumpstarter consists of several key components that work together to provide -testing capabilities for both physical hardware and virtual devices: - -### Device Under Test (DUT) - -The DUT is the hardware or virtual device being tested with Jumpstarter. One or -more devices can be connected to a single exporter instance so they are treated -as a single unit that can be tested together. - -### Clients - -The Jumpstarter client is a Python library and CLI tool that connects to -exporters either locally through a socket or remotely through a central server. -Clients can run test scripts, direct commands, or interactive shells to control -hardware. - -For more information, see [Clients](./introduction/clients.md). - -### Drivers - -Jumpstarter drivers are modular components that provide the ability to interact -with specific hardware interfaces or virtual devices. Drivers follow a -consistent pattern with interface definitions, implementation classes, and -client interfaces. - -For more information, see [Drivers](./introduction/drivers.md). - -### Adapters - -Adapters transform connections established by drivers into different forms or interfaces. -While drivers establish and manage the basic connections to hardware or virtual devices, -adapters take these connections and provide alternative ways to interact with them, making -them more convenient for specific use cases. - -For example, a network driver might establish a basic TCP connection, while an adapter -could transform that connection into a web-based VNC client interface, a Unix socket, -or a serial console-like interface. - -For more information, see [Adapters](./introduction/adapters.md). - -### Exporters - -The exporter is a service that runs locally or on a remote Linux device and -"exports" the interfaces connected to the Device Under Test. The exporter -implements a gRPC service that clients can connect to either directly or through -a controller to interact with devices. - -For more information, see [Exporters](./introduction/exporters.md). - -### Controller and Router - -In distributed environments, Jumpstarter provides Kubernetes-based components: - -- **Controller**: Manages client leases on exporter instances and tracks - connected clients and exporters -- **Router**: Facilitates message routing between clients and exporters through - gRPC streams - -For more information, see [Service](./introduction/service.md). - -## Operation Modes - -Jumpstarter supports two primary operation modes: local and distributed. - -### Local Mode - -In local mode, clients communicate directly with exporters running on the same -machine or through direct network connections. - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -flowchart TB - subgraph "Developer Machine" - Client["Client\n(Python Library/CLI)"] - Exporter["Exporter\n(Local Service)"] - end - - subgraph "Target Devices" - DUT["Physical/Virtual\nDevice Under Test"] - Power["Power Interface"] - Serial["Serial Interface"] - Storage["Storage Interface"] - end - - Client <--> |"gRPC via Socket"| Exporter - Exporter --> Power - Exporter --> Serial - Exporter --> Storage - Power --> DUT - Serial --> DUT - Storage --> DUT -``` - -When no client configuration or environment variables are present, Jumpstarter -runs in local mode and communicates with a built-in exporter service via a local -socket connection. - -This mode enables easy development of tests and drivers without requiring -Kubernetes or other infrastructure, whether working with physical hardware or -virtual devices. - -#### Example: Running Local Tests - -```shell -jmp shell --exporter my-hardware-exporter -jmp shell --exporter my-virtual-exporter - -pytest test_device.py -``` - -### Distributed Mode - -In distributed mode, a Kubernetes-based controller manages access to exporters -distributed across a network, with JWT token-based authentication securing all -connections. - -```{mermaid} -:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} -flowchart TB - subgraph "Kubernetes Cluster" - Controller["Controller\nResource Management"] - Router["Router\nMessage Routing"] - Auth["Authentication\nJWT Tokens"] - end - - subgraph "Test Runners" - Client1["Client 1\n(CI Pipeline)"] - Client2["Client 2\n(Developer)"] - end - - subgraph "Lab Resources" - Exporter1["Exporter 1\n(Physical Hardware)"] - Exporter2["Exporter 2\n(Virtual Devices)"] - subgraph "Devices" - DUT1["Physical Device 1"] - DUT2["Physical Device 2"] - DUT3["Virtual Device"] - end - end - - Client1 <--> |"JWT Authentication"| Auth - Client2 <--> |"JWT Authentication"| Auth - Exporter1 <--> |"JWT Authentication"| Auth - Exporter2 <--> |"JWT Authentication"| Auth - Auth <--> Controller - - Client1 <--> |"gRPC (Authorized)"| Controller - Client2 <--> |"gRPC (Authorized)"| Controller - Controller <--> Router - Router <--> |"gRPC"| Exporter1 - Router <--> |"gRPC"| Exporter2 - Exporter1 --> DUT1 - Exporter1 --> DUT2 - Exporter2 --> DUT3 -``` - -This mode supports multiple clients and exporters, with the controller managing -leases to ensure exclusive access to both hardware and virtual device resources. - -#### Authentication Flow - -The authentication flow in distributed mode works as follows: - -1. **Client Registration**: Clients are registered in the Kubernetes cluster - with a unique identity -2. **Token Issuance**: The controller issues JWT tokens to authenticated clients - and exporters -3. **Secure Communication**: All gRPC communication between components is - secured with these tokens -4. **Access Control**: The controller enforces permissions based on token - identity: - - - Which exporters a client can request leases for - - What actions a client can perform - - Which driver packages are allowed to be loaded - -This authentication mechanism enables fine-grained access control in multi-user -environments and prevents unauthorized access to hardware resources. - -#### Example: Running Distributed Tests - -```shell -# Configure client with server information -jmp config client use my-client - -# Request a lease on an exporter with specific labels -jmp create lease --selector vendor=acme,model=widget-v2 - -# Run tests using the leased exporter -pytest test_device.py -``` - -## Authentication - -Authentication for both clients and exporters is handled through JWT tokens -managed by the controller. - -This authentication mechanism enables dynamic registration of clients and -exporters, allowing controlled access to hardware resources. For example, a CI -pipeline can be granted access to only specific exporters based on its -credentials. - -## Integration with Existing Tools - -Jumpstarter is designed to integrate with a wide range of existing tools and -workflows. For detailed information about integration patterns and solution -architectures, see the [Solution Architecture](./solution-architecture.md) -document. \ No newline at end of file diff --git a/docs/source/cli/clients.md b/docs/source/cli/clients.md deleted file mode 100644 index 8c0e390b4..000000000 --- a/docs/source/cli/clients.md +++ /dev/null @@ -1,94 +0,0 @@ -# Manage Clients - -The `jmp admin` CLI can be used to manage your client configurations on -the distributed service. - -## Creating a Client - -If you have configured [a Service](../introduction/service.md) and -you have a kubeconfig, the [`jmp admin` -CLI](reference/jmp-admin.md#jmp-admin-cli-reference) will attempt to use your -current credentials to provision the client automatically and produce a client -configuration file. - -You can also use the following options to specify the kubeconfig and context to use: - -- `--kubeconfig` - Set the location of your kubeconfig file. -- `--namespace` - The namespace to search in (default is `default`) - -To create a new client and its associated config, run the following command: - -```shell -$ jmp admin create client john --namespace jumpstarter-lab --unsafe -o john.yaml -``` - -This creates a client named `john` and outputs the configuration to a YAML file -named `john.yaml`: - -```yaml -apiVersion: jumpstarter.dev/v1alpha1 -kind: ClientConfig -metadata: - namespace: jumpstarter-lab - name: john -endpoint: grpc.jumpstarter.192.168.1.10.nip.io:8082 -token: <> -grpcConfig: - # please refer to the https://grpc.github.io/grpc/core/group__grpc__arg__keys.html documentation - grpc.keepalive_time_ms: 20000 -tls: - ca: '' - insecure: False -drivers: - allow: [] - unsafe: True -``` - -We use the `--unsafe` setting that configures the `drivers` section to allow any -driver packages on the client. - -```{warning} -The drivers configuration is an important security consideration. When a client connects to an exporter, -the client-side Python modules for drivers are dynamically loaded. If you don't fully trust the exporter's -configuration, you should carefully restrict which driver packages are allowed to load on the client. -``` - -A `tls` section is also included, which allows you to specify a custom CA -certificate to use for the connection, or to disable TLS verification if your -system is using self-signed certificates. - -### Manual Provisioning - -1. Apply the YAML to your cluster: - - ```yaml - # my-client.yaml - apiVersion: jumpstarter.dev/v1alpha1 - kind: Client - metadata: - name: my-client - ``` - - ```shell - $ kubectl apply -f my-client.yaml - ``` - -2. Retrieve the created client resource information: - - ```shell - $ kubectl get client my-client -o yaml - $ kubectl get client my-client -o=jsonpath='{.status.endpoint}' - $ kubectl get secret $(kubectl get client my-client -o=jsonpath='{.status.credential.name}') -o=jsonpath='{.data.token}' | base64 -d - ``` - -3. Store these credentials securely as a CI secret or distribute them to the appropriate end user. - - The end user can then configure their client using the - [jmp](./reference/jmp.md#jmp-cli-reference) CLI: - - ```shell - $ jmp config client create my-client - Enter a valid Service endpoint: devl.jumpstarter.dev - Enter a Jumpstarter auth token (hidden): *** - Enter a comma-separated list of allowed driver packages (optional): - ``` diff --git a/docs/source/cli/exporters.md b/docs/source/cli/exporters.md deleted file mode 100644 index c9bb3ac09..000000000 --- a/docs/source/cli/exporters.md +++ /dev/null @@ -1,81 +0,0 @@ -# Manage Exporters - -The `jmp admin` admin CLI can be used to manage your exporter configurations on -the distributed service. - -## Creating a exporter - -If you have configured [a Service](../introduction/service.md) and -you have a kubeconfig, the [`jmp admin` -CLI](./reference/jmp-admin.md#jmp-admin-cli-reference) will attempt to use your -current credentials to provision the client automatically, and produce a base -exporter configuration file. - -To connect a target device to Jumpstarter, an exporter instance must be -registered. - -Exporter creation must be done by an administrator user who has access to the -Kubernetes cluster where the `jumpstarter-controller` service is hosted. - -```shell -# Create the exporter instance -$ jmp admin create exporter my-exporter --namespace jumpstarter-lab -o my-exporter.yaml -``` - -This creates an exporter named `my-exporter` and produces a YAML configuration -file `my-exporter.yaml`: -```yaml -apiVersion: jumpstarter.dev/v1alpha1 -kind: ExporterConfig -metadata: - namespace: jumpstarter-lab - name: my-exporter -endpoint: grpc.jumpstarter.example.com:443 -token: <> - -``` - -`my-exporter.yaml` should be configured with the desired exported drivers -filling up the export section. - -### Example configuration - -If you don't have the hardware ready yet but you want to try things out you can -setup the exporter with something like the following example which will provide -a few mock interfaces to play with: - -```yaml -apiVersion: jumpstarter.dev/v1alpha1 -kind: ExporterConfig -metadata: - namespace: default - name: demo -endpoint: grpc.jumpstarter.192.168.1.10.nip.io:8082 -token: <> -tls: - ca: '' - insecure: True -export: - storage: - type: jumpstarter_driver_opendal.driver.MockStorageMux - power: - type: jumpstarter_driver_power.driver.MockPower - echonet: - type: jumpstarter_driver_network.driver.EchoNetwork - can: - type: jumpstarter_driver_can.driver.Can - config: - channel: 1 - interface: "virtual" -``` - -Once the exporter configuration is ready it should be installed in the exporter -host machine at `/etc/jumpstarter/exporters/my-exporter.yaml`. - -```{note} -Remember, the exporter is a Linux service that exports the interfaces to the target DUT(s) -(serial ports, video interfaces, bluetooth, anything that Jumpstarter has a driver for, -and the exporter service can reach via linux device or network). In this case the exporter -service calls back to the Service to report the available interfaces and -waits for commands. -``` \ No newline at end of file diff --git a/docs/source/cli/index.md b/docs/source/cli/index.md deleted file mode 100644 index 454e717a2..000000000 --- a/docs/source/cli/index.md +++ /dev/null @@ -1,19 +0,0 @@ -# Command-Line Interface - -This section covers the Jumpstarter command-line tools that simplify management and operation. Whether you need to interact with clients, exporters, or run tests, these CLI tools provide a consistent interface. - -The following pages will guide you through the CLI tools: - -* **[Reference](reference/index.md)** - Comprehensive command reference -* **[Clients](clients.md)** - Using client-related CLI commands -* **[Exporters](exporters.md)** - Managing exporters via CLI -* **[Run Tests](run-tests.md)** - Running tests with the CLI - -```{toctree} -:maxdepth: 1 -:hidden: -reference/index.md -clients.md -exporters.md -run-tests.md -``` diff --git a/docs/source/cli/reference/index.md b/docs/source/cli/reference/index.md deleted file mode 100644 index 4386933a2..000000000 --- a/docs/source/cli/reference/index.md +++ /dev/null @@ -1,15 +0,0 @@ -# CLI Reference - -This section provides comprehensive documentation for the Jumpstarter command-line interface tools. You'll find details on command syntax, options, and usage examples for effectively managing Jumpstarter components. - -The following pages will guide you through the CLI references: - -* **[JMP Admin](jmp-admin.md)** - Administration commands for exporters and clients (requires kubeconfig) -* **[JMP](jmp.md)** - Core commands for client and exporter interaction - -```{toctree} -:maxdepth: 1 -:hidden: -jmp-admin.md -jmp.md -``` diff --git a/docs/source/cli/reference/jmp-admin.md b/docs/source/cli/reference/jmp-admin.md deleted file mode 100644 index 01f54647c..000000000 --- a/docs/source/cli/reference/jmp-admin.md +++ /dev/null @@ -1,7 +0,0 @@ -# jmp-admin-cli-reference - -```{eval-rst} -.. click:: jumpstarter_cli_admin:admin - :prog: jmp-admin - :nested: full -``` diff --git a/docs/source/cli/run-tests.md b/docs/source/cli/run-tests.md deleted file mode 100644 index 5710b629a..000000000 --- a/docs/source/cli/run-tests.md +++ /dev/null @@ -1,58 +0,0 @@ -# Running tests - -## Running tests through a central server - -When a client configuration is present, Jumpstarter uses the specified endpoint -and token to authenticate with that server - -### Configuration - -By default the libraries and CLI will look for a -`~/.config/jumpstarter/client.yaml` file, which contains the endpoint and token -to authenticate with the Service. - -Alternatively the client can receive the endpoint and token as environment -variables: - -```shell -export JMP_ENDPOINT=jumpstarter.my-lab.com:1443 -export JMP_TOKEN=dGhpc2lzYXRva2VuLTEyMzQxMjM0MTIzNEyMzQtc2Rxd3Jxd2VycXdlcnF3ZXJxd2VyLTEyMzQxMjM0MTIz -``` - -This is useful for CI/CD systems that inject the environment variables into the -pipeline. - -## Running tests locally (without a server) - -When no client configuration or environment variables are set, the client will -run in local mode and create an exporter instance to interact with the hardware. - -Communication between the local client and exporter take place over a local -socket provided by `$JUMPSTARTER_HOST`. - -``` -$ jmp shell --exporter-config /exporter-config.yaml -$$ echo $JUMPSTARTER_HOST -$$ j ... -$$ exit -$ -``` - -## Using jumpstarter from testing frameworks - -### Pytest - -Jumpstarter provides a pytest base class that can be used to run tests, the base -class will attempt to: - -1. Use a local connection based on the `JUMPSTARTER_HOST` environment variable -2. Use an existing lease based on the `JMP_LEASE` environment variable, and - existing credentials. See the CLI reference for [jmp create - lease](./reference/jmp.md#jmp-cli-reference). -3. Request a lease based on the `selector` provided in the test class. - -```{eval-rst} -.. autoclass:: jumpstarter_testing.pytest.JumpstarterTest - :members: -``` - diff --git a/docs/source/conf.py b/docs/source/conf.py index 1336dd32c..a38639f8d 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -42,7 +42,7 @@ suppress_warnings = [ "ref.class", # suppress unresolved Python class references (external references - # are warnings otherwise) + # are warnings otherwise) ] # -- Options for HTML output ------------------------------------------------- diff --git a/docs/source/config/cli.md b/docs/source/config/cli.md deleted file mode 100644 index 8afc624e5..000000000 --- a/docs/source/config/cli.md +++ /dev/null @@ -1,329 +0,0 @@ -# CLI Configuration - -Jumpstarter can be configured as either a client, an exporter, or both depending -on your use case and deployment strategy. - -By default, a local client and exporter session are automatically initialized -when running test scripts through `jmp shell` command. This allows for easy -testing of new drivers, client libraries, or verifying that tests work on your -local bench. - -When interacting with remote exporters in a CLI or CI environment, multiple -clients can be configured on the same system and used to interact with different -devices and different exporter instances. - -Similarly, multiple exporters can be configured on a single host system to -interact with many different devices. However, each exporter instance is -independent, making it possible to stop, restart, and add new exporters while -others are executing tests. - -## User Configuration - -Jumpstarter stores user-specific configs in the `~/.config/jumpstarter` -directory within the user's home folder. - -The user config file defines the current client config and any user-specific -configurations for `jmp` CLI tool. - -```yaml -# ~/.config/jumpstarter/config.yaml - -apiVersion: jumpstarter.dev/v1alpha1 -kind: UserConfig -config: - current-client: default -``` - -## Clients - -Client configurations are stored in the Jumpstarter user configuration directory -`~/.config/jumpstarter/clients`. Each client config is a YAML file that contains -the client name, access token, and any configuration parameters. - -### Client Config - -```yaml -# ~/.config/jumpstarter/clients/myclient.yaml - -apiVersion: jumpstarter.dev/v1alpha1 -kind: Client -metadata: - name: myclient - namespace: jumpstarter-lab -tls: - insecure: false - ca: "" -endpoint: "jumpstarter.my-lab.com:1443" -token: "dGhpc2lzYXRva2VuLTEyMzQxMjM0MTIzNEyMzQtc2Rxd3Jxd2VycXdlcnF3ZXJxd2VyLTEyMzQxMjM0MTIz" -drivers: - # Wildcards are supported - allow: ["jumpstarter_drivers_*", "vendorpackage.*"] -``` - -- `endpoint` - The gRPC endpoint of the Jumpstarter controller server. -- `token` - A client auth token generated by the controller. -- `drivers` - Driver client library configuration. - - `allow` - A list of allowed driver namespaces to automatically load. - - `unsafe: true` - Should all drivers be allowed to load? -- `tls` - - `insecure` - Set to `true` to disable TLS verification of the gRPC endpoint. - - `ca` - Path to a custom CA certificate file to use for TLS verification. - -## Multiple Clients - -Multiple client configs can be added to the `~/.config/jumpstarter/clients` -directory manually or by using the Jumpstarter CLI. - -Similar to kubectl, Jumpstarter allows you to switch between different client -configurations using the CLI tool. A `default` client is automatically -configured when installing the Jumpstarter Helm chart with `jmp install`. - -## Creating a client in the distributed service - -To create a new client in the distributed service you must use the `jmp admin -create` subcommand, please follow the instructions in the [Service -CLI](../cli/clients.md). - -## Importing a client created by the administrator - -Importing a new client is as simple as copying the administrator provided yaml -file to `~/.config/jumpstarter/clients/`, alternatively if we have the token and -endpoint the `jmp config client create ` command can be used to create the -config file. - -To switch between different client configs, use the `jmp config client use -` command: - -```shell -$ jmp config client use another -Using client config '/home/jdoe/.config/jumpstarter/clients/another.yaml' -``` - -All client configurations can be listed with `jmp config client list`: - -```shell -$ jmp config client list -CURRENT NAME ENDPOINT PATH -* default jumpstarter1.my-lab.com:1443 /home/jdoe/.config/jumpstarter/clients/default.yaml - myclient jumpstarter2.my-lab.com:1443 /home/jdoe/.config/jumpstarter/clients/myclient.yaml - another jumpstarter3.my-lab.com:1443 /home/jdoe/.config/jumpstarter/clients/another.yaml -``` - -Clients can also be removed using `jmp config client delete `: - -```shell -$ jmp config client delete myclient -Deleted client config '/home/jdoe/.config/jumpstarter/clients/myclient.yaml' -``` - - -## Client Environment Variables - -The client configuration may also be provided by environment variables which may -be useful in CI or when writing a script that uses Jumpstarter. - -- `JUMPSTARTER_GRPC_INSECURE` - Set to `1` to disable TLS verification. -- `JMP_CLIENT_CONFIG` - A path to a client configuration YAML file to use. -- `JMP_CLIENT` - The name of a registered client config to use. -- `JMP_NAMESPACE` - The namespace in the jumpstarter-controller to use, like - `metadata.namespace`. -- `JMP_NAME` - The client/exporter name in the jumpstarter-controller to use, - like `metadata.name`. -- `JMP_ENDPOINT` - The gRPC endpoint of the Jumpstarter controller server -(overrides the config value). -- `JMP_TOKEN` - A client auth token generated by the controller (overrides the -config value). -- `JMP_DRIVERS_ALLOW` - A comma-separated list of allowed driver namespaces to -automatically load. Can be set to `UNSAFE` to allow unsafe loading of drivers. -- `JUMPSTARTER_FORCE_SYSTEM_CERTS` - Set to `1` to force the system CA - certificates, which is the behavior by default for all systems but MacOS (see - [bug](https://github.com/jumpstarter-dev/jumpstarter/issues/362)) -## System Configuration - -Jumpstarter stores system configs in the `/etc/jumpstarter` directory. This -configuration directory is primarily used by the exporters -`/etc/jumpstarter/exporters` as they often run as daemon services on the host -system. - -## Exporters - -Exporter configurations are by default stored globally in the Jumpstarter config -directory `/etc/jumpstarter/exporters`. Each exporter config is a YAML file that -provides connection details and a list of exporter drivers. - -### Exporter Config - -Exporters can be configured using a pure-YAML format, which allows for the -configuration of built-in drivers and any additional driver packages registered -by the user. - -```yaml -# /etc/jumpstarter/exporters/myexporter.yaml - -apiVersion: jumpstarter.dev/v1alpha1 -kind: Exporter -metadata: - name: myexporter - namespace: jumpstarter-lab -tls: - insecure: false - ca: "" -endpoint: "jumpstarter.my-lab.com:1443" -token: "dGhpc2lzYXRva2VuLTEyMzQxMjM0MTIzNEyMzQtc2Rxd3Jxd2VycXdlcnF3ZXJxd2VyLTEyMzQxMjM0MTIz" -export: - power: - type: "jumpstarter_driver_power.driver.PduPower" - config: - host: "192.168.1.111" - port: 1234 - username: "admin" - password: "secret" - serial: - type: "jumpstarter_driver_pyserial.driver.PySerial" - config: - url: "/dev/ttyUSB0" - baudrate: 115200 - custom: - type: "vendorpackage.CustomDriver" - config: - hello: "world" -``` - -- `metadata` - Name/namespace information for the jumsptarter controller to - identify the exporter along the token. -- `endpoint` - The gRPC endpoint of the Jumpstarter controller server. -- `token` - An exporter auth token generated by the controller. -- `export` - Exporter driver configuration. -- `tls` - - `insecure` - Set to `true` to disable TLS verification of the gRPC endpoint. - - `ca` - Path to a custom CA certificate file to use for TLS verification. - -### Exporter CLI Commands - -### Creating a exporter in the distributed service - -To create a new exporter in the distributed service you must use the `jmp admin -create` command, please follow the instructions in the [Service -CLI](../cli/exporters.md). - -### Creating a exporter configuration file -To create a new exporter configuration file from a know endpoint and token the -`jmp config exporter create ` command can be used. - -```shell -$ jmp config exporter create myexporter -Endpoint: grpc.jumpstarter.my.domain.com -Token: <> -``` - -To use a specific config when starting the exporter: - -```shell -$ jmp run --exporter my-exporter -Using exporter config '/etc/jumpstarter/exporters/another/exporter.yaml' -``` - -The path to a config can also be provided: - -```shell -jmp run --exporter-config /etc/jumpstarter/exporters/another/exporter.yaml -``` - -All exporter configurations can be listed with `jmp config exporter list`: - -```shell -$ jmp config exporter list -ALIAS PATH -test-exporter-2 /etc/jumpstarter/exporters/test-exporter-2.yaml -my-exporter /etc/jumpstarter/exporters/my-exporter.yaml -``` - -Exporters can also be removed using `jmp config exporter delete `: - -```shell -$ jmp config exporter delete myexporter -Deleted exporter config '/etc/jumpstarter/exporters/myexporter.yaml' -``` - -### Exporter Environment Variables - -The exporter configuration may also be provided by environment variables which -may be useful in CI or when writing a script that uses Jumpstarter. - -- `JUMPSTARTER_GRPC_INSECURE` - Set to `1` to disable TLS verification. to use. -- `JMP_ENDPOINT` - The gRPC endpoint of the Jumpstarter controller server -(overrides the config value). -- `JMP_TOKEN` - An exporter auth token generated by the controller (overrides -the config value). -- `JMP_NAMESPACE` - The namespace in the jumpstarter-controller to use, like - `metadata.namespace`. -- `JMP_NAME` - The client/exporter name in the jumpstarter-controller to use, - like `metadata.name`. - - -## Running an Exporter - -The exporter service can be run as a container either within the same cluster -(using node affinity) or on a remote machine that has access to the cluster over -the network. - -### Running using Podman - -To run the exporter container on a test runner using Podman: - -```{code-block} bash -:substitutions: -$ sudo podman run --rm -ti --name my-exporter --net=host --privileged \ - -v /run/udev:/run/udev -v /dev:/dev -v /etc/jumpstarter:/etc/jumpstarter \ - quay.io/jumpstarter-dev/jumpstarter:{{version}} \ - jmp run --exporter my-exporter - -INFO:jumpstarter.exporter.exporter:Registering exporter with controller -INFO:jumpstarter.exporter.exporter:Currently not leased -``` - - -This will run the exporter in the foreground and allow you to observe the -behavior. Please note that we are running the exporter as root, with access to -all devices and network interfaces, this is most likely necessary for the -exporter to access the interface devices that will allow it to control the -target DUT(s). - -Once we are satisfied with the behavior we can install the exporter as a systemd -service: - -#### Running as a Service - -To run the exporter as a service podman-systemd is recommended, by using -podman-systemd you can simply create a systemd service file at -`/etc/containers/systemd/my-exporter.container` with the following content: - -```{code-block} ini -:substitutions: -[Unit] -Description=My exporter - -[Container] -ContainerName=my-exporter -Exec=/jumpstarter/bin/jmp run --exporter my-exporter -Image=quay.io/jumpstarter-dev/jumpstarter:{{version}} -Network=host -PodmanArgs=--privileged -Volume=/run/udev:/run/udev -Volume=/dev:/dev -Volume=/etc/jumpstarter:/etc/jumpstarter - -[Service] -Restart=always -StartLimitBurst=0 - -[Install] -WantedBy=multi-user.target default.target -``` - -Then enable and start the service: - -```shell -sudo systemctl daemon-reload -sudo systemctl enable --now my-exporter -``` diff --git a/docs/source/config/index.md b/docs/source/config/index.md deleted file mode 100644 index 0a2815df7..000000000 --- a/docs/source/config/index.md +++ /dev/null @@ -1,15 +0,0 @@ -# Configuration - -This section covers how to configure Jumpstarter for different environments and use cases. You'll find detailed information about available configuration options and authentication methods. - -The following pages will guide you through configuration: - -* **[CLI Configuration](cli.md)** - Command-line interface configuration options -* **[OIDC Authentication](oidc.md)** - Setting up and using OpenID Connect authentication - -```{toctree} -:maxdepth: 1 -:hidden: -cli.md -oidc.md -``` diff --git a/docs/source/config/oidc.md b/docs/source/config/oidc.md deleted file mode 100644 index 54c5c8b0d..000000000 --- a/docs/source/config/oidc.md +++ /dev/null @@ -1,298 +0,0 @@ -# OIDC Authentication - -Jumpstarter authenticates clients and exporters with internally issued JWT -tokens by default, it can also be configured to use external OpenID Connect -(OIDC) providers. - -To use OIDC with you Jumpstarter installation, set the helm value -`jumpstarter-controller.authenticationConfiguration` to a valid -`AuthenticationConfiguration` yaml configuration. - -## Examples - -### Keycloak - -Create a new keycloak client for the jumpstarter cli, set `Client ID` to -`jumpstarter-cli`, `Valid redirect URIs` to `http://localhost/callback` and -leave the remaining fields as default. Use the following snippet as -`jumpstarter-controller.authenticationConfiguration` during Jumpstarter -installation. - -```yaml -apiVersion: jumpstarter.dev/v1alpha1 -kind: AuthenticationConfiguration -jwt: -- issuer: - url: https:///realms/ # https is mandatory - certificateAuthority: # if using self-signed certificate - audiences: - - jumpstarter-cli - claimMappings: - # use user ID prefixed with "keycloak:" as username - # e.g. keycloak:example-user - username: - claim: preferred_username - prefix: "keycloak:" -``` - -Then proceed to create clients and exporters with the `jmp admin create` -commands, set their corresponding OIDC username with the `--oidc-username` flag, -e.g. `jmp admin create client test-client --oidc-username keycloak:developer-1`. -Be sure to prefix usernames with "keycloak:", as previously configured. - -Finally, instruct the users to login with the following commands - -``` -# for clients -jmp login --client --endpoint \ - --namespace --name \ - --issuer https:///realms/ -# without additional options, the users would be directed to login with the web browser -# or the username and password can be directly specified for non-interactive login - --username --password -# or a token for machine to machine authentication, useful in CI environments - --token - -# for exporters -jmp login --exporter --endpoint \ - --namespace --name \ - --issuer https:///realms/ -# --username, --password and --token are also accepted -``` - -### Dex (for authenticating with kubernetes Service Accounts) - -Initialize a self-signed CA and sign certificate for dex - -```shell -easyrsa init-pki -easyrsa --no-pass build-ca -easyrsa --no-pass build-server-full dex.dex.svc.cluster.local - -# import certificate into secret -kubectl create namespace dex -kubectl -n dex create secret tls dex-tls \ - --cert=pki/issued/dex.dex.svc.cluster.local.crt \ - --key=pki/private/dex.dex.svc.cluster.local.key -``` - -Install dex with helm - -```yaml -# dex.values.yaml -https: - enabled: true -config: - issuer: https://dex.dex.svc.cluster.local:5556 - web: - tlsCert: /etc/dex/tls/tls.crt - tlsKey: /etc/dex/tls/tls.key - storage: - type: kubernetes - config: - inCluster: true - staticClients: - - id: jumpstarter-cli - name: Jumpstarter CLI - public: true - connectors: - - name: kubernetes - type: oidc - id: kubernetes - config: - # kubectl get --raw /.well-known/openid-configuration | jq -r '.issuer' - issuer: "https://kubernetes.default.svc.cluster.local" - rootCAs: - - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt - userNameKey: sub - scopes: - - profile -volumes: - - name: tls - secret: - secretName: dex-tls -volumeMounts: - - name: tls - mountPath: /etc/dex/tls -service: - type: ClusterIP - ports: - http: - port: 5554 - https: - port: 5556 -``` - -```shell -# Ensure OIDC discovery URLs do not require authentication -kubectl create clusterrolebinding oidc-reviewer \ - --clusterrole=system:service-account-issuer-discovery \ - --group=system:unauthenticated - -helm repo add dex https://charts.dexidp.io -helm install --namespace dex --wait -f dex.values.yaml dex dex/dex -``` - -Configure Jumpstarter to trust dex by using the following snippet as -`jumpstarter-controller.authenticationConfiguration` during Jumpstarter -installation. - -```yaml -apiVersion: jumpstarter.dev/v1alpha1 -kind: AuthenticationConfiguration -jwt: - - issuer: - url: https://dex.dex.svc.cluster.local:5556 - audiences: - - jumpstarter-cli - audienceMatchPolicy: MatchAny - certificateAuthority: | - - claimMappings: - username: - claim: "name" - prefix: "dex:" -``` - -Then proceed to create clients and exporters with the `jmp admin create` -commands, set their corresponding OIDC username with the `--oidc-username` flag, -e.g. `jmp admin create exporter test-exporter --oidc-username -dex:system:serviceaccount:default:test-service-account`. Just prefix the full -service account name with "dex:", as previously configured. - -Finally, instruct the users to login with the following commands in pods -configured with proper service accounts. - -``` -# for clients -jmp login --client --endpoint \ - --namespace --name \ - --issuer https://dex.dex.svc.cluster.local:5556 \ - --connector-id kubernetes \ - --token $(cat /var/run/secrets/kubernetes.io/serviceaccount/token) - -# for exporters -jmp login --exporter --endpoint \ - --namespace --name \ - --issuer https://dex.dex.svc.cluster.local:5556 \ - --connector-id kubernetes \ - --token $(cat /var/run/secrets/kubernetes.io/serviceaccount/token) -``` - -## Reference -```yaml -# -# CAUTION: this is an example configuration. -# Do not use this for your own cluster! -# -apiVersion: jumpstarter.dev/v1alpha1 -kind: AuthenticationConfiguration -# list of authenticators to authenticate Jumpstarter users using OIDC issued JWT tokens. -jwt: -- issuer: - # URL of the provider that allows the API server to discover public signing keys. - # Only URLs that use the https:// scheme are accepted. - url: https://example.com - # discoveryURL, if specified, overrides the URL used to fetch discovery - # information instead of using "{url}/.well-known/openid-configuration". - # The exact value specified is used, so "/.well-known/openid-configuration" - # must be included in discoveryURL if needed. - # - # The "issuer" field in the fetched discovery information must match the "issuer.url" field - # in the AuthenticationConfiguration and will be used to validate the "iss" claim in the presented JWT. - # This is for scenarios where the well-known and jwks endpoints are hosted at a different - # location than the issuer (such as locally in the cluster). - # discoveryURL must be different from url if specified and must be unique across all authenticators. - discoveryURL: https://discovery.example.com/.well-known/openid-configuration - # PEM encoded CA certificates used to validate the connection when fetching - # discovery information. If not set, the system verifier will be used. - certificateAuthority: - # audiences is the set of acceptable audiences the JWT must be issued to. - # At least one of the entries must match the "aud" claim in presented JWTs. - audiences: - - my-app - - my-other-app - # this is required to be set to "MatchAny" when multiple audiences are specified. - audienceMatchPolicy: MatchAny - # rules applied to validate token claims to authenticate users. - claimValidationRules: - # A key=value pair that describes a required claim in the JWT Token. - # If set, the claim is verified to be present in the JWT Token with a matching value. - - claim: hd - requiredValue: example.com - # Instead of claim and requiredValue, you can use expression to validate the claim. - # expression is a CEL expression that evaluates to a boolean. - # all the expressions must evaluate to true for validation to succeed. - - expression: 'claims.hd == "example.com"' - # Message customizes the error message seen in the API server logs when the validation fails. - message: the hd claim must be set to example.com - - expression: 'claims.exp - claims.nbf <= 86400' - message: total token lifetime must not exceed 24 hours - claimMappings: - # username represents an option for the username attribute. - # This is the only required attribute. - username: - # JWT claim to use as the user name. - # By default sub, which is expected to be a unique identifier of the end user. - # Admins can choose other claims, such as email or name, depending on their provider. - # However, claims other than email should be prefixed to prevent naming clashes with other authenticators. - # Mutually exclusive with username.expression. - claim: "sub" - # Prefix prepended to username claims to prevent clashes with existing names (such as internal:users). - # For example, the value oidc: will create usernames like oidc:jane.doe. - # if username.claim is set, username.prefix is required. - # Explicitly set it to "" if no prefix is desired. - # Mutually exclusive with username.expression. - prefix: "" - # expression is a CEL expression that evaluates to a string. - # - # 1. If username.expression uses 'claims.email', then 'claims.email_verified' must be used in - # username.expression or extra[*].valueExpression or claimValidationRules[*].expression. - # An example claim validation rule expression that matches the validation automatically - # applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true)'. - # 2. If the username asserted based on username.expression is the empty string, the authentication - # request will fail. - # Mutually exclusive with username.claim and username.prefix. - expression: 'claims.username + ":external-user"' - # groups represents an option for the groups attribute. - groups: - # JWT claim to use as the user's group. If the claim is present it must be an array of strings. - # Mutually exclusive with groups.expression. - claim: "sub" - # Prefix prepended to group claims to prevent clashes with existing names (such as system:groups). - # For example, the value oidc: will create group names like oidc:engineering and oidc:infra. - # if groups.claim is set, groups.prefix is required. - # Explicitly set it to "" if no prefix is desired. - # Mutually exclusive with groups.expression. - prefix: "" - # expression is a CEL expression that evaluates to a string or a list of strings. - # Mutually exclusive with groups.claim and groups.prefix. - expression: 'claims.roles.split(",")' - # uid represents an option for the uid attribute. - uid: - # Mutually exclusive with uid.expression. - claim: 'sub' - # Mutually exclusive with uid.claim - # expression is a CEL expression that evaluates to a string. - expression: 'claims.sub' - # extra attributes to be added to the UserInfo object. Keys must be domain-prefix path and must be unique. - extra: - # key is a string to use as the extra attribute key. - # key must be a domain-prefix path (e.g. example.org/foo). All characters before the first "/" must be a valid - # subdomain as defined by RFC 1123. All characters trailing the first "/" must - # be valid HTTP Path characters as defined by RFC 3986. - # k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use and cannot be used. - # key must be lowercase and unique across all extra attributes. - - key: 'example.com/tenant' - # valueExpression is a CEL expression that evaluates to a string or a list of strings. - valueExpression: 'claims.tenant' - # validation rules applied to the final user object. - userValidationRules: - # expression is a CEL expression that evaluates to a boolean. - # all the expressions must evaluate to true for the user to be valid. - - expression: "!user.username.startsWith('system:')" - # Message customizes the error message seen in the API server logs when the validation fails. - message: 'username cannot used reserved system: prefix' - - expression: "user.groups.all(group, !group.startsWith('system:'))" - message: 'groups cannot used reserved system: prefix' -``` diff --git a/docs/source/getting-started/configuration/authentication.md b/docs/source/getting-started/configuration/authentication.md new file mode 100644 index 000000000..8ba98d382 --- /dev/null +++ b/docs/source/getting-started/configuration/authentication.md @@ -0,0 +1,290 @@ +# Authentication + +Jumpstarter uses internally issued JWT tokens to authenticate clients and +exporters by default. You can also configure Jumpstarter to use external OpenID +Connect (OIDC) providers. + +To use OIDC with your Jumpstarter installation: + +1. Set the helm value `jumpstarter-controller.authenticationConfiguration` to a + valid `AuthenticationConfiguration` yaml configuration +2. Configure your OIDC provider to work with Jumpstarter +3. Create users with appropriate OIDC usernames + +## Examples + +### Keycloak + +Set up Keycloak for Jumpstarter authentication: + +1. Create a new Keycloak client with these settings: + - `Client ID`: `jumpstarter-cli` + - `Valid redirect URIs`: `http://localhost/callback` + - Leave remaining fields as default + +2. Use this configuration for + `jumpstarter-controller.authenticationConfiguration` during installation: + +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: AuthenticationConfiguration +jwt: +- issuer: + url: https:///realms/ + certificateAuthority: + audiences: + - jumpstarter-cli + claimMappings: + username: + claim: preferred_username + prefix: "keycloak:" +``` + +Note, the HTTPS URL is mandatory, and you only need to include certificateAuthority when using a self-signed certificate. The username will be prefixed with "keycloak:" (e.g., keycloak:example-user). + +3. Create clients and exporters with the `jmp admin create` commands. Be sure to + prefix usernames with `keycloak:` as configured in the claim mappings: + +```shell +$ jmp admin create client test-client --oidc-username keycloak:developer-1 +``` + +4. Instruct users to log in with: + +```shell +$ jmp login --client \ + --endpoint \ + --namespace --name \ + --issuer https:///realms/ +``` + +For non-interactive login, add username and password: + +```shell +$ jmp login --client [other parameters] \ + --username \ + --password +``` + +For machine-to-machine authentication (useful in CI environments), use a token: + +```shell +$ jmp login --client [other parameters] --token +``` + +For exporters, use similar login command but with the `--exporter` flag: + +```shell +$ jmp login --exporter \ + --endpoint \ + --namespace --name \ + --issuer https:///realms/ +``` + +### Dex + +Follow these steps to set up Dex for service account authentication: + +1. Initialize a self-signed CA and sign certificate for Dex: + +```shell +$ easyrsa init-pki +$ easyrsa --no-pass build-ca +$ easyrsa --no-pass build-server-full dex.dex.svc.cluster.local +``` + +Then import the certificate into a Kubernetes secret: + +```shell +$ kubectl create namespace dex +$ kubectl -n dex create secret tls dex-tls \ + --cert=pki/issued/dex.dex.svc.cluster.local.crt \ + --key=pki/private/dex.dex.svc.cluster.local.key +``` + +2. Install Dex with Helm using the following `values.yaml`: + +```yaml +https: + enabled: true +config: + issuer: https://dex.dex.svc.cluster.local:5556 + web: + tlsCert: /etc/dex/tls/tls.crt + tlsKey: /etc/dex/tls/tls.key + storage: + type: kubernetes + config: + inCluster: true + staticClients: + - id: jumpstarter-cli + name: Jumpstarter CLI + public: true + connectors: + - name: kubernetes + type: oidc + id: kubernetes + config: + # kubectl get --raw /.well-known/openid-configuration | jq -r '.issuer' + issuer: "https://kubernetes.default.svc.cluster.local" + rootCAs: + - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + userNameKey: sub + scopes: + - profile +volumes: + - name: tls + secret: + secretName: dex-tls +volumeMounts: + - name: tls + mountPath: /etc/dex/tls +service: + type: ClusterIP + ports: + http: + port: 5554 + https: + port: 5556 +``` + +Ensure OIDC discovery URLs do not require authentication: + +```shell +$ kubectl create clusterrolebinding oidc-reviewer \ + --clusterrole=system:service-account-issuer-discovery \ + --group=system:unauthenticated +``` + +Then install Dex: + +```shell +$ helm repo add dex https://charts.dexidp.io +$ helm install --namespace dex --wait -f values.yaml dex dex/dex +``` + +3. Configure Jumpstarter to trust Dex. Use this configuration for + `jumpstarter-controller.authenticationConfiguration` during installation: + +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: AuthenticationConfiguration +jwt: + - issuer: + url: https://dex.dex.svc.cluster.local:5556 + audiences: + - jumpstarter-cli + audienceMatchPolicy: MatchAny + certificateAuthority: | + + claimMappings: + username: + claim: "name" + prefix: "dex:" +``` + +4. Create clients and exporters with appropriate OIDC usernames. Prefix the full + service account name with "dex:" as configured in the claim mappings.: + +```shell +$ jmp admin create exporter test-exporter \ + --oidc-username dex:system:serviceaccount:default:test-service-account +``` + +5. Configure pods with proper service accounts to log in using: + +For clients: + +```shell +$ jmp login --client \ + --endpoint \ + --namespace --name \ + --issuer https://dex.dex.svc.cluster.local:5556 \ + --connector-id kubernetes \ + --token $(cat /var/run/secrets/kubernetes.io/serviceaccount/token) +``` + +For exporters: + +```shell +$ jmp login --exporter \ + --endpoint \ + --namespace --name \ + --issuer https://dex.dex.svc.cluster.local:5556 \ + --connector-id kubernetes \ + --token $(cat /var/run/secrets/kubernetes.io/serviceaccount/token) +``` + +## Reference + +The reference section provides a complete example of an +`AuthenticationConfiguration` resource with detailed comments. Use this as a +template for creating your own configuration. + +Key components include: + +- JWT issuer configuration +- Claim validation rules +- Claim mappings for username and groups +- User validation rules + +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: AuthenticationConfiguration +# JWT authenticators for OIDC-issued tokens +jwt: +- issuer: + # URL of the OIDC provider (must use https://) + url: https://example.com + # Optional: override URL for discovery information + discoveryURL: https://discovery.example.com/.well-known/openid-configuration + # Optional: PEM encoded CA certificates for validation + certificateAuthority: + # List of acceptable token audiences + audiences: + - my-app + - my-other-app + # Required when multiple audiences are specified + audienceMatchPolicy: MatchAny + # rules applied to validate token claims to authenticate users. + claimValidationRules: + # Validate specific claim values + - claim: hd + requiredValue: example.com + # Alternative: use CEL expressions for complex validation + - expression: 'claims.hd == "example.com"' + message: the hd claim must be set to example.com + - expression: 'claims.exp - claims.nbf <= 86400' + message: total token lifetime must not exceed 24 hours + # Map OIDC claims to Jumpstarter user properties + claimMappings: + # Required: configure username mapping + username: + # JWT claim to use as username + claim: "sub" + # Prefix for username (required when claim is set) + prefix: "" + # Alternative: use CEL expression (mutually exclusive with claim+prefix) + # expression: 'claims.username + ":external-user"' + # Optional: configure groups mapping + groups: + claim: "sub" + prefix: "" + # Alternative: use CEL expression + # expression: 'claims.roles.split(",")' + # Optional: configure UID mapping + uid: + claim: 'sub' + # Alternative: use CEL expression + # expression: 'claims.sub' + # Optional: add extra attributes to UserInfo + extra: + - key: 'example.com/tenant' + valueExpression: 'claims.tenant' + # validation rules applied to the final user object. + userValidationRules: + - expression: "!user.username.startsWith('system:')" + message: 'username cannot used reserved system: prefix' + - expression: "user.groups.all(group, !group.startsWith('system:'))" + message: 'groups cannot used reserved system: prefix' +``` \ No newline at end of file diff --git a/docs/source/getting-started/configuration/files.md b/docs/source/getting-started/configuration/files.md new file mode 100644 index 000000000..a479f62ec --- /dev/null +++ b/docs/source/getting-started/configuration/files.md @@ -0,0 +1,158 @@ +# Files + +This page describes configuration files used in Jumpstarter, including their +format, location, related environment variables, and management commands. + +Jumpstarter follows a specific hierarchy when loading configurations. See +[Loading Order](loading-order.md) for details on how configurations from +different sources are prioritized. + +## User Configuration + +**File**: `config.yaml` +**Location**: `~/.config/jumpstarter/config.yaml` +**Description**: Defines global user settings including current client +selection. + +**Format**: +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: UserConfig +config: + current-client: default +``` + +**CLI Commands**: Modified through `jmp config` commands. + +## Client Configuration + +**File**: Various files with `.yaml` extension +**Location**: `~/.config/jumpstarter/clients/*.yaml` +**Description**: Stores client configurations including endpoints, access +tokens, and driver settings. + +**Format**: +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: Client +metadata: + name: myclient + namespace: jumpstarter-lab +tls: + insecure: false + ca: "" +endpoint: "jumpstarter.my-lab.com:1443" +token: "dGhpc2lzYXRva2VuLTEyMzQxMjM0MTIzNEyMzQtc2Rxd3Jxd2VycXdlcnF3ZXJxd2VyLTEyMzQxMjM0MTIz" +drivers: + allow: ["jumpstarter_drivers_*", "vendorpackage.*"] +``` + +**Environment Variables**: +- `JUMPSTARTER_GRPC_INSECURE` - Set to `1` to disable TLS verification +- `JMP_CLIENT_CONFIG` - Path to a client configuration file +- `JMP_CLIENT` - Name of a registered client config +- `JMP_NAMESPACE` - Namespace in the controller +- `JMP_NAME` - Client name +- `JMP_ENDPOINT` - gRPC endpoint (overrides config file) +- `JMP_TOKEN` - Auth token (overrides config file) +- `JMP_DRIVERS_ALLOW` - Comma-separated list of allowed driver namespaces +- `JUMPSTARTER_FORCE_SYSTEM_CERTS` - Set to `1` to force system CA certificates + +**CLI Commands**: +```shell +jmp config client create # Create new client config +jmp config client use # Switch to a different client +jmp config client list # List available clients +jmp config client delete # Remove a client config +``` + +## Exporter Configuration + +**File**: Various files with `.yaml` extension +**Location**: `/etc/jumpstarter/exporters/*.yaml` +**Description**: Defines exporter settings including connection details and +driver configurations. + +**Format**: +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: Exporter +metadata: + name: myexporter + namespace: jumpstarter-lab +tls: + insecure: false + ca: "" +endpoint: "jumpstarter.my-lab.com:1443" +token: "dGhpc2lzYXRva2VuLTEyMzQxMjM0MTIzNEyMzQtc2Rxd3Jxd2VycXdlcnF3ZXJxd2VyLTEyMzQxMjM0MTIz" +export: + power: + type: "jumpstarter_driver_power.driver.PduPower" + config: + host: "192.168.1.111" + port: 1234 + username: "admin" + password: "secret" + serial: + type: "jumpstarter_driver_pyserial.driver.PySerial" + config: + url: "/dev/ttyUSB0" + baudrate: 115200 +``` + +**Environment Variables**: +- `JUMPSTARTER_GRPC_INSECURE` - Set to `1` to disable TLS verification +- `JMP_ENDPOINT` - gRPC endpoint (overrides config file) +- `JMP_TOKEN` - Auth token (overrides config file) +- `JMP_NAMESPACE` - Namespace in the controller +- `JMP_NAME` - Exporter name + +**CLI Commands**: +```shell +jmp config exporter create # Create new exporter config +jmp config exporter list # List available exporters +jmp config exporter delete # Remove an exporter config +``` + +## Running Exporters + +Exporters can be run manually or as system services: + +```shell +# Run with specific exporter config +jmp run --exporter my-exporter + +# Or specify a config path directly +jmp run --exporter-config /etc/jumpstarter/exporters/my-exporter.yaml +``` + +For persistent operation, exporters can be installed as systemd services using +podman-systemd. Create a systemd service file at +`/etc/containers/systemd/my-exporter.container` with the following content: + +```{code-block} ini +:substitutions: +[Unit] +Description=My exporter +[Container] +ContainerName=my-exporter +Exec=/jumpstarter/bin/jmp run --exporter my-exporter +Image=quay.io/jumpstarter-dev/jumpstarter:{{version}} +Network=host +PodmanArgs=--privileged +Volume=/run/udev:/run/udev +Volume=/dev:/dev +Volume=/etc/jumpstarter:/etc/jumpstarter +[Service] +Restart=always +StartLimitBurst=0 +[Install] +WantedBy=multi-user.target default.target +``` + +Then enable and start the service: + +```shell +sudo systemctl daemon-reload +sudo systemctl enable --now my-exporter +``` \ No newline at end of file diff --git a/docs/source/getting-started/configuration/index.md b/docs/source/getting-started/configuration/index.md new file mode 100644 index 000000000..ca1548784 --- /dev/null +++ b/docs/source/getting-started/configuration/index.md @@ -0,0 +1,21 @@ +# Configuration + +This section explains how to configure Jumpstarter for your environment. + +- [Files](files.md): Understanding the structure and + content of configuration files +- [Loading Order](loading-order.md): Understanding how + configuration files are prioritized from different sources (environment + variables, command line, system and user config files) +- [Authentication](authentication.md): Setting up OIDC and managing tokens + +For a list of supported configuration options including an explanation please +refer to the [MAN pages](../../reference/man-pages/index.md). + +```{toctree} +:maxdepth: 1 +:hidden: +files.md +loading-order.md +authentication.md +``` \ No newline at end of file diff --git a/docs/source/getting-started/configuration/loading-order.md b/docs/source/getting-started/configuration/loading-order.md new file mode 100644 index 000000000..6a5a097af --- /dev/null +++ b/docs/source/getting-started/configuration/loading-order.md @@ -0,0 +1,74 @@ +# Loading Order + +Jumpstarter uses a hierarchical approach to loading configuration, allowing you +to override settings at different levels. + +## Configuration Sources + +Jumpstarter loads configuration from the following sources, in order of +precedence (highest to lowest): + +1. **Command-line arguments** - Highest priority, override all other settings +2. **Environment variables** - Override file-based configurations +3. **User configuration files** - Located in + `${HOME}/.config/jumpstarter/` +4. **System configuration files** - Located in `/etc/jumpstarter/` + +## Client Configuration Hierarchy + +For client operations, Jumpstarter processes configurations in this order: + +1. **Command-line options** such as `--endpoint` or `--client-config` +2. **Environment variables** such as `JMP_ENDPOINT`, `JMP_TOKEN`, or + `JMP_CLIENT_CONFIG` +3. **Current client** defined in + `${HOME}/.config/jumpstarter/config.yaml` +4. **Specific client file** in + `${HOME}/.config/jumpstarter/clients/.yaml` + +## Exporter Configuration Hierarchy + +For exporter operations, Jumpstarter processes configurations in this order: + +1. **Command-line options** such as `--exporter` or `--exporter-config` +2. **Environment variables** such as `JMP_ENDPOINT`, `JMP_TOKEN`, or + `JMP_NAMESPACE` +3. **Specific exporter file** in `/etc/jumpstarter/exporters/.yaml` + +## Example + +Here's a practical example of how configuration overrides work: + +1. You create a client configuration file at + `${HOME}/.config/jumpstarter/clients/default.yaml`: + + ```yaml + endpoint: "jumpstarter1.my-lab.com:1443" + ``` + +2. You set an environment variable in your terminal: + + ```shell + $ export JMP_ENDPOINT="jumpstarter2.my-lab.com:1443" + ``` + +3. You run a command with an explicit endpoint argument: + + ```shell + $ jmp --endpoint jumpstarter3.my-lab.com:1443 info + ``` + +Jumpstarter connects to `jumpstarter3.my-lab.com:1443` because the command-line +argument has the highest priority. + +## Use Cases + +Choose the appropriate configuration method based on your needs: + +- **Development**: Use user config files for personal settings +- **CI/CD Pipelines**: Use environment variables for automation +- **One-off Tasks**: Use command-line arguments for temporary changes +- **System Defaults**: Use system config files for shared settings across users + +This hierarchical approach allows Jumpstarter to be flexible across different +usage scenarios while maintaining consistent behavior. \ No newline at end of file diff --git a/docs/source/getting-started/index.md b/docs/source/getting-started/index.md index c2220e115..56e9bbe99 100644 --- a/docs/source/getting-started/index.md +++ b/docs/source/getting-started/index.md @@ -1,15 +1,22 @@ -# Getting Started Guides +# Getting Started -This section provides guides to help you set up Jumpstarter and test your software on real hardware. Whether you're a beginner or an experienced user, these guides will help you get up and running quickly. +This section provides hands-on guides to start using Jumpstarter in your own +environment. The guides cover: -The following pages will guide you through the setup process: +- [Installation](installation/index.md): Setting up Jumpstarter packages and + services +- [Configuration](configuration/index.md): Configuring clients, exporters, and + authentication +- [Usage](usage/index.md): Running your first tests and integrating with your + development workflow -* **[Setup Local Exporter](setup-local-exporter.md)** - Setting up Jumpstarter with a local exporter -* **[Setup Exporter Client](setup-exporter-client.md)** - Configuring and using the exporter client +These guides support both local-mode for individual development and +distributed-mode for team environments with shared hardware resources. ```{toctree} :maxdepth: 1 :hidden: -setup-local-exporter.md -setup-exporter-client.md -``` +installation/index.md +configuration/index.md +usage/index.md +``` \ No newline at end of file diff --git a/docs/source/getting-started/installation/index.md b/docs/source/getting-started/installation/index.md new file mode 100644 index 000000000..edddcb930 --- /dev/null +++ b/docs/source/getting-started/installation/index.md @@ -0,0 +1,14 @@ +# Installation + +This section provides guidance on installing Jumpstarter components in your +environment. The guides cover: + +- [Packages](packages.md): Installing Jumpstarter software packages +- [Service](service.md): Setting up Jumpstarter as a Kubernetes service + +```{toctree} +:maxdepth: 1 +:hidden: +packages.md +service.md +``` \ No newline at end of file diff --git a/docs/source/getting-started/installation/packages.md b/docs/source/getting-started/installation/packages.md new file mode 100644 index 000000000..46a23d453 --- /dev/null +++ b/docs/source/getting-started/installation/packages.md @@ -0,0 +1,95 @@ +# Packages + +## Python + +Jumpstarter includes the following installable Python packages: + +- `jumpstarter`: Core package for exporter interaction and service hosting +- `jumpstarter-cli`: CLI components metapackage including admin and user + interfaces +- `jumpstarter-cli-admin`: Admin CLI for controller management and lease control +- `jumpstarter-driver-*`: Drivers for device connectivity +- `jumpstarter-imagehash`: Image checking library for video inputs +- `jumpstarter-testing`: Tools for Jumpstarter-powered pytest integration + +### Installing Release Packages + +The [Jumpstarter Python packages](https://pkg.jumpstarter.dev/) provide all the +tools you need to run an exporter or interact with your hardware as a client. + +Install the Python package using `pip` or a similar tool. You need Python +{{requires_python}}: + +```shell +$ pip3 install --extra-index-url https://pkg.jumpstarter.dev/ jumpstarter-all +$ mkdir -p "${HOME}/.config/jumpstarter/" +$ sudo mkdir /etc/jumpstarter +``` + +The command above installs packages globally. For library usage, we recommend +installing in a virtual environment instead: + +```shell +$ python3 -m venv ~/.venv/jumpstarter +$ source ~/.venv/jumpstarter/bin/activate +$ pip3 install --extra-index-url https://pkg.jumpstarter.dev/ jumpstarter-all +``` + +### Installing from Source + +Jumpstarter undergoes active development with frequent feature additions. We +conduct thorough testing and recommend installing the latest version from the +`main` branch. + +```shell +$ sudo dnf install -y uv make git +$ git clone https://github.com/jumpstarter-dev/jumpstarter.git +$ cd jumpstarter +$ rm .python-version +$ make sync +$ mkdir -p "${HOME}/.config/jumpstarter/" +$ sudo mkdir /etc/jumpstarter +``` + +Activate the virtual environment to use Jumpstarter CLI commands: + +```shell +$ source .venv/bin/activate +$ jmp version +``` + +### Running in a Container + +If you prefer not to install packages locally, you can use the container package +instead. To interact with the service without local Python package installation, +create an alias to run the `jmp` client in a container. We recommend adding this +alias to your shell profile for persistent use: + +```{code-block} shell +:substitutions: +$ alias jmp='podman run --rm -it -w /home \ + -v "$(pwd):/home":z \ + -v "${HOME}/.config/jumpstarter/:/root/.config/jumpstarter":z \ + quay.io/jumpstarter-dev/jumpstarter:{{version}} jmp' +``` + +When you need hardware access for running the `jmp` command or following the +[local-only workflow](../../introduction/index.md#local-mode), configure the +container with device access, host networking, and privileged mode. This +typically requires `root` privileges: + +```{code-block} shell +:substitutions: +$ mkdir -p "${HOME}/.config/jumpstarter/" /etc/jumpstarter +$ alias jmp='podman run --rm -it \ + -v "${HOME}/.config/jumpstarter/:/root/.config/jumpstarter":z \ + --net=host --privileged \ + -v /run/udev:/run/udev -v /dev:/dev -v /etc/jumpstarter:/etc/jumpstarter:z \ + quay.io/jumpstarter-dev/jumpstarter:{{version}} jmp' +``` + +If you've configured a `jmp` alias you can undefine it with: + +```shell +$ unalias jmp +``` \ No newline at end of file diff --git a/docs/source/getting-started/installation/service.md b/docs/source/getting-started/installation/service.md new file mode 100644 index 000000000..60662f542 --- /dev/null +++ b/docs/source/getting-started/installation/service.md @@ -0,0 +1,275 @@ +# Service + +This section explains how to install and configure the Jumpstarter service in +your Kubernetes cluster. The service enables centralized management of your +Jumpstarter environment. Before installing, ensure you have: + +- A Kubernetes cluster available +- `kubectl` installed and configured to access your cluster +- [Helm](https://helm.sh/docs/intro/install/) (version 3.x or newer) +- Administrator access to your cluster (required for CRD installation) +- Domain name for service endpoints (or use `nip.io` for local testing) + +```{note} +`global.baseDomain` creates these service hostnames with `jumpstarter.example.com`: +- `grpc.jumpstarter.example.com` +- `router.jumpstarter.example.com` (for router endpoints) +``` + +## Kubernetes with Helm + +Install Jumpstarter on a standard Kubernetes cluster using Helm: + +```{code-block} bash +:substitutions: +$ helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ + --create-namespace --namespace jumpstarter-lab \ + --set global.baseDomain=jumpstarter.example.com \ + --set global.metrics.enabled=true \ + --set jumpstarter-controller.grpc.mode=ingress \ + --version={{controller_version}} +``` + +## OpenShift with Helm + +Install Jumpstarter on an OpenShift cluster using Helm: + +```{code-block} bash +:substitutions: +$ helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ + --create-namespace --namespace jumpstarter-lab \ + --set global.baseDomain=jumpstarter.example.com \ + --set global.metrics.enabled=true \ + --set jumpstarter-controller.grpc.mode=route \ + --version={{controller_version}} +``` + +## OpenShift with ArgoCD + +First, create and label a namespace for Jumpstarter: + +```shell +$ kubectl create namespace jumpstarter-lab +$ kubectl label namespace jumpstarter-lab argocd.argoproj.io/managed-by=openshift-gitops +``` + +For ArgoCD to manage Jumpstarter CRDs, create this ClusterRole and +ClusterRoleBinding: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + annotations: + argocds.argoproj.io/name: openshift-gitops + argocds.argoproj.io/namespace: openshift-gitops + name: openshift-gitops-argocd-appcontroller-crd +rules: +- apiGroups: + - 'apiextensions.k8s.io' + resources: + - 'customresourcedefinitions' + verbs: + - '*' +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + annotations: + argocds.argoproj.io/name: openshift-gitops + argocds.argoproj.io/namespace: openshift-gitops + name: openshift-gitops-argocd-appcontroller-crd +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: openshift-gitops-argocd-appcontroller-crd +subjects: +- kind: ServiceAccount + name: openshift-gitops-argocd-application-controller + namespace: openshift-gitops +``` + +Create an ArgoCD Application to deploy Jumpstarter: + +```{warning} +The secrets `jumpstarter-controller.controllerSecret` and `jumpstarter-controller.routerSecret` +must be unique for each installation. While Helm can auto-generate these, ArgoCD cannot - +you must manually create these in your Jumpstarter namespace. +``` + +```{code-block} yaml +:substitutions: +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: jumpstarter + namespace: openshift-gitops +spec: + destination: + name: in-cluster + namespace: jumpstarter-lab + project: default + source: + chart: jumpstarter + helm: + parameters: + - name: global.baseDomain + value: devel.jumpstarter.dev + - name: global.metrics.enabled + value: "true" + - name: jumpstarter-controller.controllerSecret + value: "pick-a-secret-DONT-USE-THIS-DEFAULT" + - name: jumpstarter-controller.routerSecret + value: "again-pick-a-secret-DONT-USE-THIS-DEFAULT" + - name: jumpstarter-controller.grpc.mode + value: "route" + repoURL: quay.io/jumpstarter-dev/helm + targetRevision: "{{controller_version}}" +``` + +## Local cluster with Minikube + +Minikube runs local Kubernetes clusters using VMs or container "nodes". It works +across several platforms and supports different hypervisors, making it ideal for +local development and testing. + +Find more information on the [minikube +website](https://minikube.sigs.k8s.io/docs/start/). + +### Create a minikube cluster + +Expand the default NodePort range to include the Jumpstarter ports: + +```shell +$ minikube start --extra-config=apiserver.service-node-port-range=8000-9000 +``` + +### Install Jumpstarter with the CLI + +The Jumpstarter CLI's `jmp admin install` command simplifies installation in +your Kubernetes cluster. + +Use the minikube IP address when installing with the CLI: + +```shell +$ jmp admin install --ip $(minikube ip) +``` + +For complete documentation of the `jmp admin install` command and all available +options, see the [MAN pages](../../reference/man-pages/jmp.md). + +### Install Jumpstarter with Helm + +For manual installation with Helm, use these commands: + +```{code-block} shell +:substitutions: +$ export IP=$(minikube ip) +$ export BASEDOMAIN="jumpstarter.${IP}.nip.io" +$ export GRPC_ENDPOINT="grpc.${BASEDOMAIN}:8082" +$ export GRPC_ROUTER_ENDPOINT="router.${BASEDOMAIN}:8083" +$ helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ + --create-namespace --namespace jumpstarter-lab \ + --set global.baseDomain=${BASEDOMAIN} \ + --set jumpstarter-controller.grpc.endpoint=${GRPC_ENDPOINT} \ + --set jumpstarter-controller.grpc.routerEndpoint=${GRPC_ROUTER_ENDPOINT} \ + --set global.metrics.enabled=false \ + --set jumpstarter-controller.grpc.nodeport.enabled=true \ + --set jumpstarter-controller.grpc.nodeport.port=8082 \ + --set jumpstarter-controller.grpc.nodeport.routerPort=8083 \ + --set jumpstarter-controller.grpc.mode=nodeport \ + --version={{controller_version}} +``` + +## Local cluster with kind + +To try the Jumpstarter Controller on your local machine, run a local Kubernetes +cluster for development purposes. + +kind is a tool for running local Kubernetes clusters using Podman or Docker +container "nodes". + +```{tip} +Consider minikube for environments requiring [untrusted certificates](https://minikube.sigs.k8s.io/docs/handbook/untrusted_certs/). +``` + +Find more information on the [kind +website](https://kind.sigs.k8s.io/docs/user/quick-start/). + +### Create a kind cluster + +First, create a kind cluster config that enables nodeports to host the Services. +Save this as `kind_config.yaml`: + +```yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +kubeadmConfigPatches: +- | + kind: ClusterConfiguration + apiServer: + extraArgs: + "service-node-port-range": "3000-32767" +- | + kind: InitConfiguration + nodeRegistration: + kubeletExtraArgs: + node-labels: "ingress-ready=true" +nodes: +- role: control-plane + extraPortMappings: + - containerPort: 80 + hostPort: 5080 + protocol: TCP + - containerPort: 30010 + hostPort: 8082 + protocol: TCP + - containerPort: 30011 + hostPort: 8083 + protocol: TCP + - containerPort: 443 + hostPort: 5443 + protocol: TCP +``` + +Next, create a kind cluster using the config you created: + +```shell +$ kind create cluster --config kind_config.yaml +``` + +### Install Jumpstarter with the CLI + +The Jumpstarter CLI provides the `jmp admin install` command to automatically +run Helm with the correct arguments, simplifying installation in your Kubernetes +cluster. + +Install Jumpstarter with default options: + +```shell +$ jmp admin install +``` + +For complete documentation of the `jmp admin install` command and all available +options, see the [MAN pages](../../reference/man-pages/jmp.md). + +### Install Jumpstarter with Helm + +If you prefer manual installation with Helm, use the following commands: + +```{code-block} bash +:substitutions: +$ export IP="X.X.X.X" +$ export BASEDOMAIN="jumpstarter.${IP}.nip.io" +$ export GRPC_ENDPOINT="grpc.${BASEDOMAIN}:8082" +$ export GRPC_ROUTER_ENDPOINT="router.${BASEDOMAIN}:8083" +$ helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ + --create-namespace --namespace jumpstarter-lab \ + --set global.baseDomain=${BASEDOMAIN} \ + --set jumpstarter-controller.grpc.endpoint=${GRPC_ENDPOINT} \ + --set jumpstarter-controller.grpc.routerEndpoint=${GRPC_ROUTER_ENDPOINT} \ + --set global.metrics.enabled=false \ + --set jumpstarter-controller.grpc.nodeport.enabled=true \ + --set jumpstarter-controller.grpc.mode=nodeport \ + --version={{controller_version}} +``` \ No newline at end of file diff --git a/docs/source/getting-started/setup-exporter-client.md b/docs/source/getting-started/setup-exporter-client.md deleted file mode 100644 index 5ee02f2ce..000000000 --- a/docs/source/getting-started/setup-exporter-client.md +++ /dev/null @@ -1,144 +0,0 @@ -# Setup a Remote Exporter/Client - -This guide walks you through the process of creating an exporter using the -controller service, configuring drivers, and running the exporter. - -## Prerequisites - -Make sure [the following packages are -installed](../installation/python-package.md) in your Python environment: -- `jumpstarter-cli` - The core Jumpstarter CLI -- `jumpstarter-driver-opendal` - The OpenDAL storage driver -- `jumpstarter-driver-power` - The base power driver - -You should also have the [Service](../introduction/service.md) -running in a Kubernetes cluster you have admin access to. For instructions on -how to install Jumpstarter in a cluster, refer to the [installation -guide](../installation/service/index.md). - -```{tip} -Make sure you have the correct cluster in your `kubeconfig` file and the right -context selected. -``` - -## Create an Exporter - -First, we must create an exporter using the controller service API. The `jmp -admin` CLI provides methods to interact with the controller directly. - -To create an exporter and save the configuration locally, run the following command: - -```shell -# Creates an exporter called "testing" and saves the config -$ jmp admin create exporter testing --save -``` - -## Usage for jmp admin create exporter - -```{command-output} jmp admin create exporter --help -``` - -### Edit the Exporter Configuration - -Once the exporter has been created, a new configuration file will be saved to -`/etc/jumpstarter/exporters/testing.yaml`. - -To edit the configuration file with your default text editor, run the following -command: - -```shell -# Opens the config for "testing" in your default editor -$ jmp config exporter edit testing -``` - -Add the `storage` and `power` drivers under the `export` field in the configuration -file. The finished configuration should look like this: - -```yaml -# /etc/jumpstarter/exporters/testing.yaml -apiVersion: jumpstarter.dev/v1alpha1 -kind: ExporterConfig -# These values are automatically filled by the controller -endpoint: "..." -token: "..." -# Mock drivers for demo purpose -export: - storage: - type: jumpstarter_driver_opendal.driver.MockStorageMux - power: - type: jumpstarter_driver_power.driver.MockPower -``` - -## Run an Exporter - -To run the exporter locally, we can use the `jmp` CLI tool. - -Run the following command to start the exporter locally using the configuration file: - -```shell -# Runs the exporter "testing" locally -$ jmp run --exporter testing -``` - -The exporter will continue running until the process is terminated via `^C` or the shell -is closed. - -## Create a Client - -To connect to the new exporter, a client must be created. We can also use the -`jmp admin` CLI tool to create a client using the controller. - -```shell -# This will create a client called "hello", allow unsafe drivers, and save the config -$ jmp admin create client hello --save --unsafe -``` - -## Usage for jmp admin create client - -```{command-output} jmp admin create client --help -``` - -## Connect to the Exporter - -To interact with the exporter we created above, we can use the "client shell" -functionality within the `jmp` CLI. When a shell is spawned, the client attempts -to acquire a lease on an exporter. Once the lease is acquired, the client can be -interacted with through the magic `j` command or via the Python API. - -```shell -# Spawn a shell using the "hello" client -$ jmp shell --client hello --selector example.com/board=foo -``` - -## Usage for jmp shell - -```{command-output} jmp shell --help -``` - -Once a lease is acquired, we can interact with the drivers hosted by the -exporter within the shell instance. - -```shell -# Spawn a shell using the "hello" client -$ jmp shell --client hello --selector example.com/board=foo - -# Running inside client shell -$ j -Usage: j [OPTIONS] COMMAND [ARGS]... - - Generic composite device - -Options: - --help Show this message and exit. - -Commands: - power Generic power - storage Generic storage mux - -# Simulate turning on the power -$ j power on -ok - -# Exit the shell -$ exit -``` diff --git a/docs/source/getting-started/setup-local-exporter.md b/docs/source/getting-started/setup-local-exporter.md deleted file mode 100644 index 891f34fbe..000000000 --- a/docs/source/getting-started/setup-local-exporter.md +++ /dev/null @@ -1,184 +0,0 @@ -# Setup a Local Exporter - -This guide walks you through the process of using Jumpstarter with a local -exporter (i.e., the client and the exporter running on the same host). - -## Prerequisites - -Make sure the following packages are installed in your Python environment: -- `jumpstarter-cli` - The core Jumpstarter CLI -- `jumpstarter-driver-opendal` - The OpenDAL storage driver -- `jumpstarter-driver-power` - The base power driver - -```{tip} -Both of these driver packages provide mock implementations, making it easier -to debug the connection between an exporter and client without hardware. -``` - -## Create an Exporter Config - -First, we must create an exporter configuration to define the "shape" of the exporter -that we are going to test locally. This configuration is identical to a regular -exporter config; however, the `endpoint` and `token` fields may be left empty as -we do not need to connect to the controller service. - -Create a text file in `/etc/jumpstarter/exporters` with the following content: - -```{note} -The name of this file is used when referring to the exporter in later steps. -``` - -```yaml -# /etc/jumpstarter/exporters/demo.yaml -apiVersion: jumpstarter.dev/v1alpha1 -kind: ExporterConfig -metadata: - namespace: default - name: demo -# endpoint and token are intentionally left empty -endpoint: "" -token: "" -# mock drivers for demo purpose -export: - storage: - type: jumpstarter_driver_opendal.driver.MockStorageMux - power: - type: jumpstarter_driver_power.driver.MockPower -``` - -## Spawn an Exporter Shell - -To interact locally with the exporter we created above, we can use the "exporter -shell" functionality within the `jmp` CLI. When a shell is spawned, a local -exporter instance runs in the background while the shell session is active. - -```shell -# Spawn a new exporter shell for "demo" -$ jmp shell --exporter demo -``` - -### Interact with the Exporter Shell - -If the drivers specified in the exporter configuration provide a CLI interface, it will -be available through the magic `j` command within the exporter shell. - -```shell -# Enter the shell -$ jmp shell --exporter demo - -# Running inside exporter shell -$ j -Usage: j [OPTIONS] COMMAND [ARGS]... - - Generic composite device - -Options: - --help Show this message and exit. - -Commands: - power Generic power - storage Generic storage mux - -# Simulate turning on the power -$ j power on -ok - -# Exit the shell -$ exit -``` - -### Use the Python API in a Shell - -As the shell exposes the local exporter via environment variables, we can run -any command or Python script that interacts with a client/exporter. - -This allows us to use the Python API directly without having to interact with -the CLI. This is often useful for more complex scripts or if a specific driver -doesn't provide a client CLI. - -#### Running a Python Script - -The easiest way to interact with the exporter is to run a quick Python script -directly from the command line. This is particularly useful when no CLI is available. - -```shell -# Enter the shell -$ jmp shell --exporter demo -# Running python inside exporter shell -$ python - < `, where available actions depend on the specific driver. + +### Use the Python API in a Shell + +The exporter shell exposes the local exporter via environment variables, +enabling you to run any Python code that interacts with the client/exporter. +This approach works especially well for complex operations or when a driver +doesn't provide a CLI. + +#### Using Python with Jumpstarter + +Create a Python file for interacting with your exporter. This example +(`example.py`) demonstrates a complete power cycle workflow: + +```python +import time +from jumpstarter.common.utils import env + +with env() as client: + client.power.on() + client.power.off() +``` + +```shell +$ jmp shell # Use appropriate --exporter or --client parameters +$ python ./example.py +$ exit +``` + +This example demonstrates how Python interacts with the exporter: + +1. The `env()` function from `jumpstarter.common.utils` automatically connects + to the exporter configured in your shell environment. + +2. The `with env() as client:` statement creates a client connected to your + local exporter and handles connection setup and cleanup. + +3. `client.power.on()` directly calls the power driver's "on" methodโ€”the same + action that `j power on` performs in the CLI. + +4. `client.power.off()` directly calls the power driver's "off" methodโ€”the same + action that `j power off` performs in the CLI. + +Using a Python with Jumpstarter allows you to: + + - Create sequences of operations (power on โ†’ wait โ†’ power off) + - Save and reuse complex workflows + - Add logic, error handling, and conditional operations + - Import other Python libraries (like `time` in this example) + - Build sophisticated automation scripts + +#### Running `pytest` in the Shell + +For multiple test cases, run a `pytest` suite using Jumpstarter's built-in +testing library as defined in `example_test.py`: + +```python +from jumpstarter_testing.pytest import JumpstarterTest + +class MyTest(JumpstarterTest): + def test_power_on(self, client): + client.power.on() + + def test_power_off(self, client): + client.power.off() +``` + +```shell +$ jmp shell # Use appropriate --exporter or --client parameters +$ pytest ./example_test.py +$ exit +``` + +This example demonstrates using `pytest` for structured testing with +Jumpstarter: + +1. The `JumpstarterTest` is a `pytest` fixture that: + + - Automatically establishes a connection to your exporter + - Provides a pre-configured `client` object to each test method + - Handles setup and teardown between tests + +2. Each test method receives the `client` parameter, giving access to all driver + interfaces just like in the previous examples. + +Benefits of using `pytest` with Jumpstarter are: + + - Organize tests into logical classes and methods + - Generate test reports with success/failure statuses + - Use `pytest`'s extensive features (parameterization, fixtures, etc.) + - Run selective tests based on names or tags \ No newline at end of file diff --git a/docs/source/getting-started/usage/index.md b/docs/source/getting-started/usage/index.md new file mode 100644 index 000000000..12a8c246a --- /dev/null +++ b/docs/source/getting-started/usage/index.md @@ -0,0 +1,23 @@ +# Usage + +This section provides guidance on how to use Jumpstarter effectively in your +development workflow. The guides cover: + +- [Setup Local Mode](setup-local-mode.md): Running Jumpstarter in local mode for + individual development +- [Setup Distributed Mode](setup-distributed-mode.md): Configuring Jumpstarter + for team environments with shared resources +- [Examples](examples.md): Practical examples of Jumpstarter usage in common + scenarios +- [Integration Patterns](integration-patterns.md): Integrate Jumpstarter into + your existing workflows and systems + + +```{toctree} +:maxdepth: 1 +:hidden: +setup-local-mode.md +setup-distributed-mode.md +examples.md +integration-patterns.md +``` \ No newline at end of file diff --git a/docs/source/solution-architecture.md b/docs/source/getting-started/usage/integration-patterns.md similarity index 85% rename from docs/source/solution-architecture.md rename to docs/source/getting-started/usage/integration-patterns.md index b0d37a341..b36a03f80 100644 --- a/docs/source/solution-architecture.md +++ b/docs/source/getting-started/usage/integration-patterns.md @@ -1,20 +1,16 @@ -# Solution Architecture +# Integration Patterns -This document outlines common solution architectures and integration patterns -for Jumpstarter, helping you understand how to incorporate it into your -development and testing workflows. +This document outlines common integration patterns for Jumpstarter, helping you +incorporate it into your development and testing workflows. -## Common Integration Patterns +Jumpstarter integrates with various tools and platforms across the hardware +development lifecycle: -Jumpstarter is designed to integrate with various tools and platforms across the -hardware development lifecycle. Here are the key integration points: - -- **CI/CD Systems**: Jenkins, GitHub Actions, GitLab CI, Tekton -- **Infrastructure Tools**: Kubernetes, Prometheus, Grafana -- **Development Workflows**: IDEs, shell scripts, Makefiles +- **Infrastructure**: Kubernetes, Prometheus, Grafana +- **Developer Environments**: IDE, scripts, GitHub Actions, GitLab CI, Tekton - **Testing Frameworks**: pytest, unittest, Robot Framework -## Integration with Infrastructure +## Infrastructure ### Continuous Integration with System Testing @@ -114,10 +110,10 @@ flowchart TB GitRepo -- "Code changes" --> Actions Actions -- "Dispatch job" --> Runner1 - + Runner1 -- "Execute tests" --> JmpLocal JmpLocal -- "Control" --> Devices - + Runner1 -- "Report results" --> Actions Actions -- "Update status" --> GitRepo ``` @@ -130,9 +126,11 @@ This architecture leverages a self-hosted runner with directly attached system: 4. Tests execute on the runner using Jumpstarter to interface with the system 5. Results are reported back to the CI system -This approach is ideal when: -- System needs to be permanently connected to a specific test machine -- You want to integrate system testing into existing CI/CD workflows without additional infrastructure +This approach works best when: + +- You need to permanently connect systems to a specific test machine +- You want to integrate system testing into existing CI/CD workflows without + additional infrastructure - You need a simple setup for initial system-in-the-loop testing **CI Configuration Examples:** @@ -182,27 +180,28 @@ hardware-test: ### Cost Management and Chargeback -Organizations can implement usage-based billing for teams through a cost management layer. +Organizations can implement usage-based billing for teams through a cost +management layer. ```{mermaid} :config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} flowchart LR subgraph "Kubernetes" Controller["Controller"] - + subgraph "Telemetry" Prometheus["Prometheus"] Grafana["Grafana"] AlertManager["AlertManager"] end - + subgraph "Cost Management" UsageTracker["Usage Tracker"] OpenCost["OpenCost"] Accounting["Chargeback System"] end end - + subgraph "Lab" Rack1["Exporter 1"] Rack2["Exporter 2"] @@ -218,30 +217,32 @@ flowchart LR Controller -- "Connect to" --> Rack1 Controller -- "Connect to" --> Rack2 - + Rack1 -- "Report usage\nmetrics" --> Prometheus Rack2 -- "Report usage\nmetrics" --> Prometheus - + Prometheus -- "Store\nmetrics" --> Grafana Prometheus -- "Threshold\nalerts" --> AlertManager Prometheus -- "Usage\nmetrics" --> UsageTracker - + UsageTracker -- "Monthly billing\nreport" --> Team - + UsageTracker -- "Team resource\nusage" --> OpenCost OpenCost -- "Cost\nallocation" --> Accounting ``` -This architecture implements a cost chargeback model for infrastructure resources: +This architecture implements a cost chargeback model for infrastructure +resources: -1. Prometheus serves as the foundation for collecting and storing all resource utilization metrics -2. The team requests resources through the controller, which records team identifiers with each lease +1. Prometheus collects and stores all resource utilization metrics +2. Teams request resources through the controller, which records team + identifiers with each lease 3. System resources export detailed utilization metrics to Prometheus: - Resource uptime and availability - Utilization metrics (CPU, memory, I/O) - Team attribution via metadata -## Integration with Developer Environments +## Developer Environments ### Traditional Developer Workflow @@ -262,7 +263,7 @@ flowchart TB RemoteExporters["Exporter"] LabDevices["Device Under Test"] end - + TestCode --> LocalExporter LocalExporter --> DeviceOnDesk @@ -272,15 +273,18 @@ flowchart TB RemoteExporters --> LabDevices ``` -This architecture supports developers working with both local systems and -shared lab resources: +This architecture supports developers working with both local systems and shared +lab resources: -1. Developers write code and test code in their IDE +1. Developers write and test code in their IDE 2. For quick tests, they use the test code to access a system on their desk 3. For more complex tests, they connect to remote lab systems through the controller 4. The same test code works in both environments +See [Setup Local Mode](setup-local-mode.md) for more information on configuring +your local environment. + ### Cloud Native Developer Workflow ```{mermaid} @@ -289,67 +293,80 @@ flowchart TB subgraph "Web Browser" Dev["Developer"] end - + subgraph "Kubernetes Cluster" subgraph "Eclipse Che" Workspace["Developer Workspace"] TestCode["Test Code"] PortFwd["Port Forwarding"] end - + Controller["Controller"] end - + subgraph "Local Environment" LocalExporter["Local Exporter"] DeviceOnDesk["Device Under Test"] end - + subgraph "Lab" RemoteExporters["Exporter"] LabDevices["Device Under Test"] end - + Dev -- "Access via browser" --> Workspace Workspace -- "Contains" --> TestCode - + TestCode -- "Local system access" --> PortFwd PortFwd -- "Forward connection" --> LocalExporter LocalExporter -- "Control" --> DeviceOnDesk - + TestCode -- "Request access" --> Controller Controller -- "Assign lease" --> TestCode Controller -- "Connect to" --> RemoteExporters RemoteExporters -- "Control" --> LabDevices ``` -This architecture provides a modern cloud-native development experience while maintaining flexibility to work with both local and remote systems: +This architecture provides a cloud-native development experience while +maintaining flexibility to work with both local and remote systems: -1. Developers access a containerized development environment through a web browser using Eclipse Che -2. The development workspace contains all necessary tools, dependencies, and test code +1. Developers access a containerized development environment through a web + browser using Eclipse Che +2. The development workspace contains all necessary tools, dependencies, and + test code 3. For quick iterations with locally connected systems: - - Port forwarding enables the cloud workspace to communicate with systems connected to the developer's machine + - Port forwarding enables the cloud workspace to communicate with systems + connected to the developer's machine - The local Jumpstarter exporter manages the device directly 4. For access to shared lab resources: - - The same test code can request access to remote devices through the controller - - The controller manages leases and routes connections through the standard infrastructure + - The same test code can request access to remote devices through the + controller + - The controller manages leases and routes connections through the standard + infrastructure Key benefits of this approach: -- **Consistent Development Environment**: Standardized, reproducible workspaces for all team members +- **Consistent Development Environment**: Standardized, reproducible workspaces + for all team members - **Flexibility**: Seamless transition between local and remote system testing -- **Collaboration**: Web-based IDE enables real-time collaboration and knowledge sharing -- **Scalability**: Easy onboarding of new team members with zero local configuration -- **System Flexibility**: Enables a hybrid approach where developers can test locally first, then validate on shared lab systems +- **Collaboration**: Web-based IDE enables real-time collaboration and knowledge + sharing +- **Scalability**: Easy onboarding of new team members with zero local + configuration +- **System Flexibility**: Enables a hybrid approach where developers can test + locally first, then validate on shared lab systems -This workflow eliminates the distinction between local and cloud development while providing the best of both worlds for system testing. +This workflow eliminates the distinction between local and cloud development +while providing the best of both worlds for system testing. -## Integration with Testing Frameworks +See [Setup Distributed Mode](setup-distributed-mode.md) for more details on +configuring your distributed environment. + +## Testing Frameworks ### pytest Integration -Jumpstarter integrates seamlessly with pytest through the `jumpstarter-testing` -package: +Jumpstarter integrates with pytest through the `jumpstarter-testing` package: ```python from jumpstarter_testing.pytest import JumpstarterTest @@ -381,7 +398,8 @@ Device Boot Test Should Not Be Empty ${output} Power Off ``` -## Best Practices + +## Recommended Practices ### Labeling Strategy @@ -395,7 +413,7 @@ selection straightforward: ### Resource Management -Implement practices to ensure efficient use of shared systems: +Implement these practices to ensure efficient use of shared systems: - Set appropriate lease timeouts to prevent orphaned resources - Use CI systems' concurrency controls to manage test parallelism diff --git a/docs/source/getting-started/usage/setup-distributed-mode.md b/docs/source/getting-started/usage/setup-distributed-mode.md new file mode 100644 index 000000000..78b6690c7 --- /dev/null +++ b/docs/source/getting-started/usage/setup-distributed-mode.md @@ -0,0 +1,107 @@ +# Setup Distributed Mode + +This guide walks you through the process of creating an exporter using the +controller service, configuring drivers, and running the exporter. + +## Prerequisites + +Install [the following packages](../installation/packages.md) in your Python +environment: + +- `jumpstarter-cli` - The core Jumpstarter CLI +- `jumpstarter-driver-opendal` - The OpenDAL storage driver +- `jumpstarter-driver-power` - The base power driver + +These driver packages include mock implementations, enabling you to test the +connection between an exporter and client without physical hardware. + +You need the [service](../../introduction/service.md) running in a Kubernetes +cluster with admin access. For installation instructions, refer to the +[installation guide](../installation/service.md). + +## Instructions + +### Create an Exporter Configuration + +Create an exporter using the controller service API. The `jmp admin` CLI +provides commands to interact with the controller directly. + +Run this command to create an exporter named `example-distributed` and save the +configuration locally: + +```shell +$ jmp admin create exporter example-distributed --save +``` + +After creating the exporter, find the new configuration file at +`/etc/jumpstarter/exporters/example-distributed.yaml`. Edit the configuration +using your default text editor with: + +```shell +$ jmp config exporter edit example-distributed +``` + +Add the `storage` and `power` drivers under the `export` field in the +configuration file. Your configuration should look like this: + +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: ExporterConfig +metadata: + namespace: default + name: example-distributed +endpoint: "" +token: "" +export: + storage: + type: jumpstarter_driver_opendal.driver.MockStorageMux + power: + type: jumpstarter_driver_power.driver.MockPower +``` + +### Run an Exporter + +Start the exporter locally using the `jmp` CLI tool: + +```shell +$ jmp run --exporter example-distributed +``` + +The exporter runs until you terminate the process with or close the shell. + +### Create a Client + +Create a client to connect to your new exporter using the `jmp admin` CLI: + +The following command creates a client named "hello", enables unsafe drivers for +development purposes, and saves the configuration locally in +`${HOME}/.config/jumpstarter/clients/`: + +```shell +$ jmp admin create client hello --save --unsafe +``` + +### Spawn an Exporter Shell + +Interact with your distributed exporter using the "client shell" functionality +in the `jmp` CLI. When you spawn a shell, the client attempts to acquire a lease +on an exporter. Once the lease is acquired, you can interact with the exporter +through your shell session. + +```shell +$ jmp shell --client hello --selector example.com/board=foo +``` + +### Exiting the Exporter Shell + +To terminate the local exporter, simply exit the shell: + +```shell +$ exit +``` + +## Next Steps + +Once you have your exporter shell running, you can start using Jumpstarter +commands to interact with your hardware. To learn more about common workflow +patterns and implementation examples, see [Examples](./examples.md). \ No newline at end of file diff --git a/docs/source/getting-started/usage/setup-local-mode.md b/docs/source/getting-started/usage/setup-local-mode.md new file mode 100644 index 000000000..7df226be1 --- /dev/null +++ b/docs/source/getting-started/usage/setup-local-mode.md @@ -0,0 +1,66 @@ +# Setup Local Mode + +This guide shows you how to use Jumpstarter with a client and exporter running +on the same host. + +## Prerequisites + +Install [the following packages](../installation/packages.md) in your Python +environment: + +- `jumpstarter-cli` - The Jumpstarter CLI for interacting with exporters +- `jumpstarter-driver-opendal` - The OpenDAL storage driver for file operations +- `jumpstarter-driver-power` - The base power driver for managing power states + +These driver packages include mock implementations, enabling you to test the +connection between an exporter and client without physical hardware. + +## Instructions + +### Create an Exporter Configuration + +Create an exporter configuration named `example-local` to define the +capabilities of your local test exporter. This configuration mirrors a regular +exporter config but leaves the `endpoint` and `token` fields empty since you +don't need to connect to the controller service. + +Create `example-local.yaml` in `/etc/jumpstarter/exporters` with this content: + +```yaml +apiVersion: jumpstarter.dev/v1alpha1 +kind: ExporterConfig +metadata: + namespace: default + name: example-local +endpoint: "" +token: "" +export: + storage: + type: jumpstarter_driver_opendal.driver.MockStorageMux + power: + type: jumpstarter_driver_power.driver.MockPower +``` + +### Spawn an Exporter Shell + +Interact with your local exporter using the "exporter shell" functionality in +the `jmp` CLI. When you spawn a shell, Jumpstarter runs a local exporter +instance in the background for the duration of your shell session. + +```shell +$ jmp shell --exporter example-local +``` + +### Exiting the Exporter Shell + +To terminate the local exporter, simply exit the shell: + +```shell +$ exit +``` + +## Next Steps + +Once you have your exporter shell running, you can start using Jumpstarter +commands to interact with your hardware. To learn more about common workflow +patterns and implementation examples, see [Examples](./examples.md). \ No newline at end of file diff --git a/docs/source/glossary.md b/docs/source/glossary.md index 06ffd72a9..167323400 100644 --- a/docs/source/glossary.md +++ b/docs/source/glossary.md @@ -3,6 +3,11 @@ ## Acronyms * `DUT`: Device Under Test +* `CRD`: Custom Resource Definition +* `CI/CD`: Continuous Integration/Continuous Deployment +* `gRPC`: Google Remote Procedure Call +* `JWT`: JSON Web Token +* `KVM`: Keyboard, Video, Mouse ## Entities @@ -16,9 +21,17 @@ exporters and clients, manages leases, and provides an inventory of available exporters and clients. +* `router`: A service used by the controller to route messages between clients + and exporters through a gRPC tunnel, enabling remote access to exported + interfaces. + +* `host`: A system running the exporter service, typically a low-cost test + system such as a single board computer with sufficient interfaces to connect + to hardware. + ## Concepts -* `Device`: A device that is exposed on an exporter. The exporter enumerates +* `device`: A device that is exposed on an exporter. The exporter enumerates these devices and makes them available for use in tests. Examples of resources include: * Network interface @@ -27,12 +40,14 @@ * Storage device (USB Muxer, SD-Wire, etc.) * CAN bus interface -* `Lease`: A time-limited reservation of an exporter. A lease is created by a +* `lease`: A time-limited reservation of an exporter. A lease is created by a client and allows the client to use the exporter resources for a limited time. + Leases ensure exclusive access to specific devices/exporters. -* `adapter`: A component that transforms connections exposed by drivers into different forms - or interfaces. Adapters take a driver client as input and provide alternative ways to interact - with the underlying connection, such as port forwarding, VNC access, or terminal emulation. +* `adapter`: A component that transforms connections exposed by drivers into + different forms or interfaces. Adapters take a driver client as input and + provide alternative ways to interact with the underlying connection, such as + port forwarding, VNC access, or terminal emulation. * `interface class`: An abstract base class that defines the contract for driver implementations. It specifies the required methods that must be implemented by @@ -43,13 +58,35 @@ `Driver` base class. It uses the `@export` decorator to expose methods that can be called remotely by clients. -* `driver client class`: The driver client class that is used - directly by end users. It interacts with the `driver class` remotely via - remote procedure call to invoke exported methods, which in turn interact with - the exporter resources. +* `driver client class`: The driver client class that is used directly by end + users. It interacts with the `driver class` remotely via remote procedure call + to invoke exported methods, which in turn interact with the exporter + resources. + +* `driver`: The term for both the `driver class` and the corresponding `driver + client class`, not to be confused with `Driver`, the base class of all `driver + classes`. Drivers in the main `jumpstarter` repository are called `in-tree + drivers`, otherwise they are called `out-of-tree drivers`. Drivers + implementing predefined interfaces are called `standard drivers`, otherwise + they are called `custom drivers`. + +* `composite driver`: A driver that combines multiple lower-level drivers to + create higher-level abstractions or specialized workflows, organized in a tree + structure to represent complex device configurations. + +* `local mode`: An operation mode where clients communicate directly with + exporters running on the same machine or through direct network connections, + ideal for individual developers working directly with accessible hardware or + virtual devices. + +* `distributed mode`: An operation mode that enables multiple teams to securely + share hardware resources across a network using a Kubernetes-based controller + to coordinate access to exporters and manage leases. + +* `stream`: A continuous data exchange channel established by drivers for + communications like serial connections or video streaming, enabling real-time + interaction with both physical and virtual interfaces across the network. -* `driver`: The term for both the `driver class` and the corresponding `driver client - class`, not to be confused with `Driver`, the base class of all `driver - classes`. Drivers in the main `jumpstarter` repository are called `in-tree drivers`, - otherwise they are called `out-of-tree drivers`. Drivers implementing predefined - interfaces are called `standard drivers`, otherwise they are called `custom drivers`. +* `message`: Commands sent from driver clients to driver implementations, + allowing the client to trigger actions or retrieve information from the + device. \ No newline at end of file diff --git a/docs/source/index.md b/docs/source/index.md index 20ed71075..d1e24e963 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -22,52 +22,41 @@ :alt: Weekly Meeting ``` -Jumpstarter is a free and open source testing tool that bridges the gap between development workflows and deployment environments. It enables you to test your software stack consistently across both real hardware and virtual environments using cloud native principles. By decoupling your target devices (physical or virtual) from test runners, development machines, and CI/CD pipelines, Jumpstarter allows you to use the same automation scripts everywhere - like a *Makefile* for device automation. +Jumpstarter is a free and open source testing tool that bridges the gap between +development workflows and deployment environments. It enables you to test your +software stack consistently across both real hardware and virtual environments +using cloud native principles. See Jumpstarter in action: + +```{raw} html + +``` + +One tool, any target. Jumpstarter decouples devices from test runners, letting +you use identical automation scripts everywhere - your *Makefile* for device +testing. ```{include} ../../README.md :start-after: "## Highlights" :end-before: "##" ``` -## Learning Paths - -### ๐Ÿ”ฐ For Newcomers -**New to Jumpstarter?** Start here to understand the core concepts and basic workflows: -- [What is Jumpstarter?](introduction/index.md) - Understand the key concepts and components -- [Installation Guide](installation/index.md) - Get Jumpstarter installed on your system -- [Setting Up Your First Local Exporter](getting-started/setup-local-exporter.md) - Connect to your first device - -### ๐Ÿ’ป For Testers & Developers -**Want to use Jumpstarter for testing?** These resources will help you automate tests for your devices: -- [Setting Up Client & Exporter](getting-started/setup-exporter-client.md) - Configure your testing environment -- [Command Line Interface](cli/index.md) - Master the CLI for automation scripts -- [Example Projects](https://github.com/jumpstarter-dev/jumpstarter/tree/main/examples) - Real-world testing examples - -### ๐Ÿ› ๏ธ For Contributors -**Looking to extend Jumpstarter?** Learn how to develop your own drivers and components: -- [Architecture Overview](architecture.md) - Understand how Jumpstarter works internally -- [Driver Development](introduction/drivers.md) - Create custom drivers for new hardware -- [API Reference](api-reference/index.md) - Comprehensive API documentation -- [Contributing Guide](contributing.md) - Guidelines for contributing to the project - -### ๐Ÿข For Teams & Enterprise -**Building a distributed CI environment?** Scale Jumpstarter across your organization: -- [Distributed Mode Setup](installation/service/index.md) - Deploy the Kubernetes-based controller -- [Solution Architecture](solution-architecture.md) - Reference architectures for complex environments -- [Managing Lab Resources](introduction/service.md) - Coordinate access to shared hardware +Ready to get started? Use the navigation menu to find documentation that fits +your needs. ```{toctree} :maxdepth: 3 :hidden: introduction/index.md -installation/index.md getting-started/index.md -cli/index.md -config/index.md -architecture.md -solution-architecture.md + contributing.md glossary.md -api-reference/index.md + +reference/index.md ``` \ No newline at end of file diff --git a/docs/source/installation/index.md b/docs/source/installation/index.md deleted file mode 100644 index d54c39705..000000000 --- a/docs/source/installation/index.md +++ /dev/null @@ -1,15 +0,0 @@ -# Installation - -This section contains guides to install the latest version of Jumpstarter components. Whether you need the Python packages or the full service deployment, you'll find all the necessary installation instructions here. - -The following pages will guide you through the installation process: - -* **[Python Package](python-package.md)** - Installing the Jumpstarter Python packages -* **[Service](service/index.md)** - Deploying the Service in Kubernetes - -```{toctree} -:maxdepth: 1 -:hidden: -python-package.md -service/index.md -``` diff --git a/docs/source/installation/python-package.md b/docs/source/installation/python-package.md deleted file mode 100644 index 8fc3d9c24..000000000 --- a/docs/source/installation/python-package.md +++ /dev/null @@ -1,135 +0,0 @@ -# Python Packages and CLI - -## Release install - -The [Jumpstarter Python packages](https://pkg.jumpstarter.dev/) -contain all the necessary tools to run an exporter or interact with your -hardware as a client. - -The Python package can be installed using ``pip`` or similar. Python -{{requires_python}} is required: - -```shell -$ pip3 install --extra-index-url https://pkg.jumpstarter.dev/ jumpstarter-all - -$ mkdir -p "${HOME}/.config/jumpstarter/" -$ sudo mkdir /etc/jumpstarter -``` - -```{tip} -This will install the `jumpstarter` packages globally. When using Jumpstarter -as a library, it is recommended to install the package in a virtual environment -instead: - -$ python3 -m venv ~/.venv/jumpstarter - -$ source ~/.venv/jumpstarter/bin/activate - -$ pip3 install .... -``` - -An alternative to installing the packages is to [use the container -package](#running-in-a-container). - -## Development install -Jumpstarter is under active development, and new features are added frequently. -We perform basic e2e testing and thorough unit testing, so we recommend -installing the latest version from the `main` branch. - -For this, you will need a few tools like `uv`, `make`, and `git`: -```shell -$ sudo dnf install -y uv make git - -# Clone the repository -$ git clone https://github.com/jumpstarter-dev/jumpstarter.git -$ cd jumpstarter - -$ rm .python-version # remove the Python version pinning - -$ make # creates the dist directory with all the packages -$ make sync # installs the packages in a local .venv - -# create the configuration directories -$ mkdir -p "${HOME}/.config/jumpstarter/" -$ sudo mkdir /etc/jumpstarter - -``` - -Then you can use the Jumpstarter CLI commands by activating the Python virtual -environment: -```shell -$ source .venv/bin/activate -$ jmp version -``` - -```{tip} -If you configured a jmp alias to use the container, -please undefine those aliases before running the `jmp` command. - -i.e. `unalias jmp` -``` - - -## Running in a Container - -For interacting with the Service without installing the Python -packages locally, you can create an alias to run the `jmp` client in a -container. - -```{tip} -It is recommended to add the alias to your shell profile. -``` - -```{code-block} bash -:substitutions: -$ alias jmp='podman run --rm -it -w /home \ - -v "$(pwd):/home":z \ - -v "${HOME}/.config/jumpstarter/:/root/.config/jumpstarter":z \ - quay.io/jumpstarter-dev/jumpstarter:{{version}} jmp' -``` - -Then you can try: - -```shell -$ jmp config client list -CURRENT NAME ENDPOINT PATH -* default grpc.devel.jumpstarter.dev:443 /root/.config/jumpstarter/clients/default.yaml - test grpc.devel.jumpstarter.dev:443 /root/.config/jumpstarter/clients/test.yaml -``` - -### Hardware Access for Exporters - -If you need access to your hardware, e.g., because you are running the `jmp` -command or you are following the [local-only -workflow](../architecture.md#local-mode) (i.e., without a distributed service), -you need to mount access to devices into the container, provide host network -access, and run the container in privileged mode. This will probably need to be run -as **root**. - - -```{code-block} bash -:substitutions: -$ mkdir -p "${HOME}/.config/jumpstarter/" /etc/jumpstarter - -# you may want to add this alias to the profile -$ alias jmp='podman run --rm -it \ - -v "${HOME}/.config/jumpstarter/:/root/.config/jumpstarter":z \ - --net=host --privileged \ - -v /run/udev:/run/udev -v /dev:/dev -v /etc/jumpstarter:/etc/jumpstarter:z \ - quay.io/jumpstarter-dev/jumpstarter:{{version}} jmp' -``` - -## Python Components - - -The Jumpstarter packages which can be installed are: - -| Component | Description | -| ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [`jumpstarter`](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter) | The core Jumpstarter Python package. This is necessary to lease and interact with the exporters; it's also the component that runs on the exporter hosts as a service. In most cases, installation is not necessary and can be consumed through another package such as `jumpstarter-cli`. | -| [`jumpstarter-cli`](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-cli) | A metapackage containing all of the Jumpstarter CLI components including the cluster admin CLI `jumpstarter-cli-admin` and the user-facing CLI. | -| [`jumpstarter-cli-admin`](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-cli-admin) | The Jumpstarter admin CLI (`jmp-admin`). This CLI can be used to install the Jumpstarter controller, manage client/exporter registrations, and monitor/control leases. | -| [`jumpstarter-driver-*`](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages) | All community and official driver packages that are distributed as part of Jumpstarter are prefixed with `jumpstarter-driver-*`. This includes drivers for PySerial, SD Wire, HTTP, CAN, and more. Driver packages only need to be installed on the exporter/client if they are used by your testing environment. All drivers are optional. | -| [`jumpstarter-adapter-*`](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages) | All community and official adapter packages that are distributed as part of Jumpstarter are prefixed with `jumpstarter-adapter-*`. This includes adapters to redirect streams to local ports, unix sockets, perform SSH connections, etc. | -| [`jumpstarter-imagehash`](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-imagehash) | A library to perform image checking from video inputs using the simple Python imagehash library | -| [`jumpstarter-testing`](https://github.com/jumpstarter-dev/jumpstarter/tree/main/packages/jumpstarter-testing) | Testing tools for writing Jumpstarter-powered tests with `pytest`. | diff --git a/docs/source/installation/service/index.md b/docs/source/installation/service/index.md deleted file mode 100644 index 4855ded7a..000000000 --- a/docs/source/installation/service/index.md +++ /dev/null @@ -1,21 +0,0 @@ -# Service Installation - -This section explains how to install and configure the Service in your Kubernetes cluster. The service enables centralized management of your Jumpstarter lab environment. You'll need a Kubernetes cluster, kubectl, and Helm to proceed with installation. - -The following pages will guide you through service installation for different environments: - -* **[Kubernetes with Helm](kubernetes-helm.md)** - Standard Kubernetes installation using Helm -* **[OpenShift with Helm](openshift-helm.md)** - Installation on OpenShift using Helm -* **[OpenShift with ArgoCD](openshift-argocd.md)** - GitOps-based installation using ArgoCD -* **[Kind with Helm](kind-helm.md)** - Installation on a local Kind cluster -* **[Minikube with Helm](minikube-helm.md)** - Installation on Minikube - -```{toctree} -:maxdepth: 1 -:hidden: -kubernetes-helm.md -openshift-helm.md -openshift-argocd.md -kind-helm.md -minikube-helm.md -``` diff --git a/docs/source/installation/service/kind-helm.md b/docs/source/installation/service/kind-helm.md deleted file mode 100644 index d44e0e65e..000000000 --- a/docs/source/installation/service/kind-helm.md +++ /dev/null @@ -1,107 +0,0 @@ -# Local cluster with KinD - -If you want to play with the Jumpstarter Controller on your local machine, we -recommend running a local Kubernetes cluster for development purposes. - -Kind is a tool for running local Kubernetes clusters using Podman or Docker -container "nodes". - -```{tip} -We recommend using [minikube](./minikube-helm.md) if you cannot easily use Kind in your local environment -(e.g. need to use [untrusted root certificates](https://minikube.sigs.k8s.io/docs/handbook/untrusted_certs/)). -``` - - -You can find more information on the [kind -website](https://kind.sigs.k8s.io/docs/user/quick-start/). - -## Installation - -### Create a kind cluster - -First, create a kind cluster config that enables the use of nodeports to host -the Services. - -```yaml -# kind_config.yaml -kind: Cluster -apiVersion: kind.x-k8s.io/v1alpha4 -kubeadmConfigPatches: -- | - kind: ClusterConfiguration - apiServer: - extraArgs: - "service-node-port-range": "3000-32767" -- | - kind: InitConfiguration - nodeRegistration: - kubeletExtraArgs: - node-labels: "ingress-ready=true" -nodes: -- role: control-plane - extraPortMappings: - - containerPort: 80 # ingress controller - hostPort: 5080 - protocol: TCP - - containerPort: 30010 # grpc nodeport - hostPort: 8082 - protocol: TCP - - containerPort: 30011 # grpc router nodeport - hostPort: 8083 - protocol: TCP - - containerPort: 443 # minimalistic UI - hostPort: 5443 - protocol: TCP -``` - -Next, create a kind cluster using the config you created. - -```shell -kind create cluster --config kind_config.yaml -``` - -### Install Jumpstarter with the CLI - -To simplify the installation in your Kubernetes cluster, the Jumpstarter CLI -provides the `jmp admin install` command to automatically run Helm with the -correct arguments. - -```{tip} -If you do not have Helm installed, please [install the latest release](https://helm.sh/docs/intro/install/). -``` - -``` -# Install Jumpstarter with default options -$ jmp admin install -``` - -## Options provided by the Jumpstarter CLI - -```{command-output} jmp admin install --help -``` - -### Install Jumpstarter with Helm - -If you prefer to manually install with Helm, the following command should work. - -```{code-block} bash -:substitutions: -# Get the IP address of your computer -# On Linux you can run: ip route get 1.1.1.1 | grep -oP 'src \K\S+' -export IP="X.X.X.X" -# Setup the base domain and endpoints with nip.io -export BASEDOMAIN="jumpstarter.${IP}.nip.io" -export GRPC_ENDPOINT="grpc.${BASEDOMAIN}:8082" -export GRPC_ROUTER_ENDPOINT="router.${BASEDOMAIN}:8083" - -# Install the Service in the namespace jumpstarter-lab with Helm -helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ - --create-namespace --namespace jumpstarter-lab \ - --set global.baseDomain=${BASEDOMAIN} \ - --set jumpstarter-controller.grpc.endpoint=${GRPC_ENDPOINT} \ - --set jumpstarter-controller.grpc.routerEndpoint=${GRPC_ROUTER_ENDPOINT} \ - --set global.metrics.enabled=false \ - --set jumpstarter-controller.grpc.nodeport.enabled=true \ - --set jumpstarter-controller.grpc.mode=nodeport \ - --version={{controller_version}} -``` diff --git a/docs/source/installation/service/kubernetes-helm.md b/docs/source/installation/service/kubernetes-helm.md deleted file mode 100644 index 635810921..000000000 --- a/docs/source/installation/service/kubernetes-helm.md +++ /dev/null @@ -1,20 +0,0 @@ -# Kubernetes - -The Service can be installed on a Kubernetes cluster using Helm. - -## Install with Helm - -```{note} -Please note that `global.baseDomain` is used to create the hostnames for the services. -With the provided example, the services will be available at grpc.jumpstarter.example.com -``` - -```{code-block} bash -:substitutions: -helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ - --create-namespace --namespace jumpstarter-lab \ - --set global.baseDomain=jumpstarter.example.com \ - --set global.metrics.enabled=true `# disable if metrics not available` \ - --set jumpstarter-controller.grpc.mode=ingress \ - --version={{controller_version}} -``` diff --git a/docs/source/installation/service/minikube-helm.md b/docs/source/installation/service/minikube-helm.md deleted file mode 100644 index b91559f66..000000000 --- a/docs/source/installation/service/minikube-helm.md +++ /dev/null @@ -1,72 +0,0 @@ -# Local cluster with minikube - -If you want to play with the Jumpstarter Controller on your local machine, we -recommend running a local Kubernetes cluster for development purposes. - -minikube is a tool for running local Kubernetes clusters using local VMs or -Podman/Docker container "nodes", it works across several platforms and can be -used with different hypervisors. - -You can find more information on the [minikube -website](https://minikube.sigs.k8s.io/docs/start/). - -## Installation - -### Start a minikube cluster - -First, we must start a local minikube cluster with the correct features enabled -to support Jumpstarter. - -```shell -# We must expand the default NodePort range to include the Jumpstarter ports -minikube start --extra-config=apiserver.service-node-port-range=8000-9000 -``` - -### Install Jumpstarter with the CLI - -To simplify the installation in your Kubernetes cluster, the Jumpstarter CLI -provides the `jmp admin install` command to automatically run Helm with the -correct arguments. - -```{tip} -If you do not have Helm installed, please [install the latest release](https://helm.sh/docs/intro/install/). -``` - -``` -# Install Jumpstarter with default options -$ jmp admin install --ip $(minikube ip) -``` - -## Options provided by the Jumpstarter CLI - -```{command-output} jmp admin install --help -``` - -### Install Jumpstarter with Helm - -```{tip} -If you do not have Helm installed, please [install the latest release](https://helm.sh/docs/intro/install/). -``` - -```{code-block} bash -:substitutions: -# Get the minikube cluster IP address -export IP=$(minikube ip) -# Setup the base domain and endpoints with nip.io -export BASEDOMAIN="jumpstarter.${IP}.nip.io" -export GRPC_ENDPOINT="grpc.${BASEDOMAIN}:8082" -export GRPC_ROUTER_ENDPOINT="router.${BASEDOMAIN}:8083" - -# Install the Service in the namespace jumpstarter-lab with Helm -helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ - --create-namespace --namespace jumpstarter-lab \ - --set global.baseDomain=${BASEDOMAIN} \ - --set jumpstarter-controller.grpc.endpoint=${GRPC_ENDPOINT} \ - --set jumpstarter-controller.grpc.routerEndpoint=${GRPC_ROUTER_ENDPOINT} \ - --set global.metrics.enabled=false \ - --set jumpstarter-controller.grpc.nodeport.enabled=true \ - --set jumpstarter-controller.grpc.nodeport.port=8082 \ - --set jumpstarter-controller.grpc.nodeport.routerPort=8083 \ - --set jumpstarter-controller.grpc.mode=nodeport \ - --version={{controller_version}} -``` diff --git a/docs/source/installation/service/openshift-argocd.md b/docs/source/installation/service/openshift-argocd.md deleted file mode 100644 index 0a467f611..000000000 --- a/docs/source/installation/service/openshift-argocd.md +++ /dev/null @@ -1,99 +0,0 @@ -# OpenShift (ArgoCD) - -## Create namespace -First, we must create a namespace for the Jumpstarter installation. This -namespace should be labeled with -`argocd.argoproj.io/managed-by=` to allow ArgoCD to -manage the resources in the namespace. - -In this case, using the default openshift-gitops ArgoCD deployment, the command -would be: -```shell -kubectl create namespace jumpstarter-lab -kubectl label namespace jumpstarter-lab argocd.argoproj.io/managed-by=openshift-gitops -``` - -## Note on CRDs - -ArgoCD needs to be able to manage the CRDs that Jumpstarter uses. This is done -by creating a ClusterRole and ClusterRoleBinding that allows the ArgoCD -application controller to manage the CRDs. - -An alternative to this is to manually create and update the CRDs that -jumpstarter uses. - -The ClusterRole & Binding to allow ArgoCD to manage the CRDs are: - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - annotations: - argocds.argoproj.io/name: openshift-gitops - argocds.argoproj.io/namespace: openshift-gitops - name: openshift-gitops-argocd-appcontroller-crd -rules: -- apiGroups: - - 'apiextensions.k8s.io' - resources: - - 'customresourcedefinitions' - verbs: - - '*' ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - annotations: - argocds.argoproj.io/name: openshift-gitops - argocds.argoproj.io/namespace: openshift-gitops - name: openshift-gitops-argocd-appcontroller-crd -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: openshift-gitops-argocd-appcontroller-crd -subjects: -- kind: ServiceAccount - name: openshift-gitops-argocd-application-controller - namespace: openshift-gitops -``` - -## Application - -```{warning} -The parameters `jumpstarter-controller.controllerSecret` and `jumpstarter-controller.routerSecret` -are security credentials used to secure authentication between clients and Jumpstarter components. -These secrets must be unique and should not be shared between installations. While Helm installation -can auto-generate values for these parameters, this mechanism does not work with ArgoCD. You must manually -create these secrets in the namespace where Jumpstarter will be installed. -``` - -```{code-block} yaml -:substitutions: -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: jumpstarter - namespace: openshift-gitops -spec: - destination: - name: in-cluster - namespace: jumpstarter-lab - project: default - source: - chart: jumpstarter - helm: - parameters: - - name: global.baseDomain - value: devel.jumpstarter.dev - - name: global.metrics.enabled - value: "true" - - name: jumpstarter-controller.controllerSecret - value: "pick-a-secret-DONT-USE-THIS-DEFAULT" - - name: jumpstarter-controller.routerSecret - value: "again-pick-a-secret-DONT-USE-THIS-DEFAULT" - - name: jumpstarter-controller.grpc.mode - value: "route" - repoURL: quay.io/jumpstarter-dev/helm - targetRevision: "{{controller_version}}" -``` - diff --git a/docs/source/installation/service/openshift-helm.md b/docs/source/installation/service/openshift-helm.md deleted file mode 100644 index 29d9872d3..000000000 --- a/docs/source/installation/service/openshift-helm.md +++ /dev/null @@ -1,25 +0,0 @@ -# OpenShift (Helm) -```{tip} -Please note that `global.baseDomain` is used to create the hostnames for the services. -With the provided example, the services will be available at grpc.jumpstarter.example.com -and router.jumpstarter.example.com. -```` - -```{note} -Please note that you will need administrator access to the cluster to install the Service, -as the installation process will install CRDs and ClusterRoles. -``` - -To install using Helm: - -```{code-block} bash -:substitutions: -helm upgrade jumpstarter --install oci://quay.io/jumpstarter-dev/helm/jumpstarter \ - --create-namespace --namespace jumpstarter-lab \ - --set global.baseDomain=jumpstarter.example.com \ - --set global.metrics.enabled=true \ - --set jumpstarter-controller.grpc.mode=route \ - --version={{controller_version}} -``` - - diff --git a/docs/source/introduction/adapters.md b/docs/source/introduction/adapters.md index 92934da0f..139f5151b 100644 --- a/docs/source/introduction/adapters.md +++ b/docs/source/introduction/adapters.md @@ -1,20 +1,158 @@ # Adapters -Adapters are specialized components in Jumpstarter that transform network connections established by drivers into different forms or interfaces that are more appropriate for specific use cases. +Jumpstarter uses adapters to transform network connections established by +drivers into different forms or interfaces that are more appropriate for +specific use cases. -## Adapter Architecture +## Architecture Adapters in Jumpstarter follow a transformation pattern where: - Adapters take a driver client as input - They transform the connection into a different interface format -- The transformed interface is exposed to the user in a way that's tailored for specific scenarios +- The transformed interface is exposed to the user in a way that's tailored for + specific scenarios -For comprehensive documentation on the adapter architecture, including detailed -patterns and examples, see the [Adapter Classes and Architecture](../api-reference/adapters.md) reference. +The architecture consists of these key components: -Unlike [Drivers](./drivers.md), which establish the foundational connections to hardware or virtual interfaces, adapters focus on providing alternative ways to interact with those connections without modifying the underlying drivers. +- **Adapter Base** - Adapters typically follow a context manager pattern using + Python's `with` statement for resource management. Each adapter takes a driver + client as input and transforms its connection. -## Types of Adapters +- **Connection Transformation** - Adapters create a new interface on top of an + existing driver connection, such as forwarding ports, providing web + interfaces, or offering terminal-like access. -* ๐Ÿ“ก [Network](../api-reference/adapters/network.md): Adapters that transform network connections and streams into different protocols or connection types. \ No newline at end of file +- **Resource Lifecycle** - Adapters handle proper setup and teardown of + resources, ensuring connections are properly established and cleaned up. + +Unlike [Drivers](drivers.md), which establish the foundational connections to +hardware or virtual interfaces, adapters focus on providing alternative ways to +interact with those connections without modifying the underlying drivers. +Adapters operate entirely on the client side and transform existing connections +rather than establishing new ones directly with hardware or virtual devices. + +## Types + +Different types of adapters serve different needs: + +- **Port Forwarding Adapters** - Convert network connections to local ports or + sockets +- **Interactive Adapters** - Provide interactive shells or console-like + interfaces +- **Protocol Adapters** - Transform connections to use different protocols + (e.g., SSH, VNC) +- **UI Adapters** - Create user interfaces for interacting with devices (e.g., + web-based VNC) + +Adapters can be composed and extended for more complex scenarios: + +- **Chaining adapters**: Use the output of one adapter as the input to another +- **Custom adapters**: Create specialized adapters for specific hardware or + software interfaces +- **Extended functionality**: Add logging, monitoring, or security features on + top of base adapters + +## Implementation Patterns + +Adapters typically implement the context manager protocol (`__enter__` and +`__exit__`) to ensure proper resource management. The general pattern is: + +1. Initialize with a driver client reference +2. Set up the transformed connection in `__enter__` +3. Return the appropriate interface (URL, address, interactive object) +4. Clean up resources in `__exit__` + +This allows adapters to be used in `with` statements for clean, deterministic +resource handling. + +When working with adapters, follow these recommended practices: + +1. **Always use context managers** (`with` statements) to ensure proper resource + cleanup and prevent resource leaks +2. **Consider security implications** when forwarding ports or providing network + access, especially when exposing services to external networks +3. **Implement proper error handling and retries** for robust connections in + unstable network environments +4. **Use appropriate timeouts** to prevent hanging connections and ensure + responsiveness +5. **Consider performance implications** for long-running connections or + high-throughput scenarios, especially in resource-constrained environments + +## Example Implementation + +```{testcode} +from contextlib import contextmanager +import socket +import threading +from typing import Tuple, Any + +class TcpPortforwardAdapter: + """ + Adapter that forwards a remote TCP port to a local TCP port. + + Args: + client: A network driver client that provides a connection + local_host: Host to bind to (default: 127.0.0.1) + local_port: Port to bind to (default: 0, which selects a random port) + + Returns: + A tuple of (host, port) when used as a context manager + """ + def __init__(self, client, local_host="127.0.0.1", local_port=0): + self.client = client + self.local_host = local_host + self.local_port = local_port + self._server = None + self._thread = None + + def __enter__(self) -> Tuple[str, int]: + # Create a socket server + self._server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self._server.bind((self.local_host, self.local_port)) + self._server.listen(5) + + # Get the actual port (if we used port 0) + self.local_host, self.local_port = self._server.getsockname() + + # Start a thread to handle connections + self._thread = threading.Thread(target=self._handle_connections, daemon=True) + self._thread.start() + + return (self.local_host, self.local_port) + + def __exit__(self, exc_type, exc_val, exc_tb): + if self._server: + self._server.close() + self._server = None + + # Thread will exit because it's a daemon + self._thread = None + + def _handle_connections(self): + while True: + try: + client_socket, _ = self._server.accept() + # For each connection, establish a connection to the remote + # and set up bidirectional forwarding + remote_conn = self.client.connect() + self._start_forwarding(client_socket, remote_conn) + except Exception: + # Server was closed or other error + break + + def _start_forwarding(self, local_socket, remote_conn): + # Set up bidirectional forwarding between local_socket and remote_conn + # Typically done with two threads, one for each direction + # Implementation details depend on the specific driver client interface + pass + + +# Example usage: +def example_usage(): + # Assuming 'client' is a network driver client + with TcpPortforwardAdapter(client, local_port=8080) as (host, port): + print(f"Service available at {host}:{port}") + # The service is now accessible at the local address + # while this context is active +``` diff --git a/docs/source/introduction/clients.md b/docs/source/introduction/clients.md index 45059420d..d419c0214 100644 --- a/docs/source/introduction/clients.md +++ b/docs/source/introduction/clients.md @@ -1,15 +1,12 @@ # Clients To interact with your target device from a development machine or through a -CI/CD pipeline, you must use a Jumpstarter client. - -The Python client can be used either as a library through a testing tool such as -[pytest](https://docs.pytest.org/en/stable/) or as a [CLI -tool](../cli/index.md). +CI/CD pipeline, you must use a Jumpstarter client. The Python client can be used +either as a library or as a [CLI tool](../reference/man-pages/index.md). ## Types of Clients -Jumpstarter supports two types of client configurations: local and remote. +Jumpstarter supports two types of client configurations: *local* and *remote*. ### Local Clients @@ -18,26 +15,18 @@ functionality to directly access your hardware. The local client will automatically use any drivers that are registered without the need for an exporter instance running in the background. -You can get started with a local client by following the [Setup a Local -Client](../getting-started/setup-local-exporter.md) guide. - ### Remote Clients When using Jumpstarter in *distributed* mode, the client must be configured to -connect to an instance of the Service that can authenticate and -route requests to the appropriate exporter instance. +connect to an instance of the Service that can authenticate and route requests +to the appropriate exporter instance. The following parameters are required to set up a remote client: + - The URL of a Service endpoint to connect to - An authentication token generated by the Service ```{note} The endpoint must be accessible from your client machine to communicate with the Service. -``` - -You can get started with a remote client by following the [Setup a Remote -Client](../getting-started/setup-exporter-client.md) guide. - -We will discuss how remote clients connect to exporters in the next section on -the [Service](./service.md). +``` \ No newline at end of file diff --git a/docs/source/introduction/drivers.md b/docs/source/introduction/drivers.md index 7a7f50569..40667cc52 100644 --- a/docs/source/introduction/drivers.md +++ b/docs/source/introduction/drivers.md @@ -4,11 +4,11 @@ Jumpstarter uses a modular driver model to build abstractions around the interfaces used to interact with target devices, both physical hardware and virtual systems. -An [Exporter](./exporters.md) uses Drivers to "export" these interfaces from a +An [Exporter](exporters.md) uses Drivers to "export" these interfaces from a host machine to the clients via [gRPC](https://grpc.io/). Drivers can be thought of as a simplified API for an interface or device type. -## Driver Architecture +## Architecture Drivers in Jumpstarter follow a client/server architecture where: @@ -17,30 +17,72 @@ Drivers in Jumpstarter follow a client/server architecture where: - Driver clients run on the client side and communicate with drivers via gRPC - Interface classes define the contract between implementations and clients -For comprehensive documentation on the driver architecture, including detailed -patterns and examples, see the [Driver Classes and Architecture](../api-reference/drivers.md) reference. +The architecture follows a pattern with these key components: -Drivers are often used with [Adapters](./adapters.md), which transform driver connections into different forms or interfaces for specific use cases. +- **Interface Class** - An abstract base class using Python's ABCMeta to define + the contract (methods and their signatures) that driver implementations must + fulfill. The interface also specifies the client class through the `client()` + class method. -## Driver Types +- **Driver Class** - Inherits from both the Interface and the base `Driver` + class, implementing the logic to configure and use hardware interfaces. Driver + methods are marked with the `@export` decorator to expose them over the + network. -The API reference of the documentation provides a complete list of all drivers, -you can find it here: [Driver API Reference](../api-reference/drivers/index.md). +- **Driver Client** - Provides a user-friendly interface that can be used by + clients to interact with the driver either locally or remotely over the + network. + +When a client requests a lease and connects to an exporter, a session is created +for all tests the client needs to execute. Within this session, the specified +`Driver` subclass is instantiated for each configured interface. These driver +instances live throughout the session's duration, maintaining state and +executing setup/teardown logic. + +On the client side, a `DriverClient` subclass is instantiated for each exported +interface. Since clients may run on different machines than exporters, +`DriverClient` classes are loaded dynamically when specified in the allowed +packages list. + +To maintain compatibility, avoid making breaking changes to interfaces. Add new +methods when needed but preserve existing signatures. If breaking changes are +required, create new interface, client, and driver versions within the same +module. + +Drivers are often used with [Adapters](adapters.md), which transform driver +connections into different forms or interfaces for specific use cases. + +## Types + +The API reference of the documentation provides a complete list of all standard +drivers, you can find it here: [Driver API +Reference](../reference/package-apis/drivers/index.md). Some categories of drivers include: -* โšก [System Control](../api-reference/drivers/index.md#system-control-drivers): Control power to devices, or general control. -* ๐Ÿ“ก [Communication](../api-reference/drivers/index.md#communication-drivers): Provide protocols for network communication, such as TCP/IP, Serial, CAN bus, etc. -* ๐Ÿ’พ [Storage And Data](../api-reference/drivers/index.md#storage-and-data-drivers): Control storage devices, such as SD cards or USB drives, and data. -* ๐Ÿ“น [Media](../api-reference/drivers/index.md#media-drivers): Provide interfaces for media capture and playback, such as video or audio. -* ๐Ÿž [Debug and Programming](../api-reference/drivers/index.md#debug-and-programming-drivers): Provide interfaces for debugging and programming devices, such as JTAG or SWD, remote flashing, emulation, etc. -* ๐Ÿ› ๏ธ [Utility](../api-reference/drivers/index.md#utility-drivers): Provide utility functions, such as shell driver commands on a exporter. + +- [System + Control](../reference/package-apis/drivers/index.md#system-control-drivers): + Control power to devices, or general control. +- [Communication](../reference/package-apis/drivers/index.md#communication-drivers): + Provide protocols for network communication, such as TCP/IP, Serial, CAN bus, + etc. +- [Data and Storage](../reference/package-apis/drivers/index.md#storage-and-data-drivers): Control + storage devices, such as SD cards or USB drives, and data. +- [Media](../reference/package-apis/drivers/index.md#media-drivers): Provide + interfaces for media capture and playback, such as video or audio. +- [Debug and + Programming](../reference/package-apis/drivers/index.md#debug-and-programming-drivers): + Provide interfaces for debugging and programming devices, such as JTAG or SWD, + remote flashing, emulation, etc. +- [Utility](../reference/package-apis/drivers/index.md#utility-drivers): Provide + utility functions, such as shell driver commands on a exporter. ### Composite Drivers -Composite drivers (`jumpstarter-driver-composite`) combine multiple lower-level -drivers to create higher-level abstractions or specialized workflows. For -example, a composite driver might coordinate power cycling, storage re-flashing, -and serial communication to automate a device initialization process. +Composite drivers combine multiple lower-level drivers to create higher-level +abstractions or specialized workflows. For example, a composite driver might +coordinate power cycling, storage re-flashing, and serial communication to +automate a device initialization process. In Jumpstarter, drivers are organized in a tree structure which allows for the representation of complex device configurations that may be found in your @@ -49,16 +91,16 @@ environment. Here's an example of a composite driver tree: ``` -MyHarness # Custom composite driver for the entire target device harness -โ”œโ”€ TcpNetwork # TCP Network driver to tunnel port 8000 -โ”œโ”€ MyPower # Custom power driver to control device power -โ”œโ”€ SDWire # SD Wire storage emulator to enable re-flash on demand -โ”œโ”€ DigitalOutput # GPIO pin control to send signals to the device -โ””โ”€ MyDebugger # Custom debugger interface composite driver - โ””โ”€ PySerial # Serial debugger with PySerial +MyHarness # Custom composite driver for the entire target device harness +โ”œโ”€ TcpNetwork # TCP Network driver to tunnel port 8000 +โ”œโ”€ MyPower # Custom power driver to control device power +โ”œโ”€ SDWire # SD Wire storage emulator to enable re-flash on demand +โ”œโ”€ DigitalOutput # GPIO pin control to send signals to the device +โ””โ”€ MyDebugger # Custom debugger interface composite driver + โ””โ”€ PySerial # Serial debugger with PySerial ``` -## Driver Configuration +## Configuration Drivers are configured using a YAML Exporter config file, which specifies the drivers to load and the parameters for each. Drivers are distributed as Python @@ -98,7 +140,7 @@ export: cpu_cores: 2 ``` -## Driver Communication +## Communication Drivers use two primary methods to communicate between client and exporter: @@ -112,7 +154,9 @@ Methods marked with the `@export` decorator are made available over the network. Drivers can establish streams for continuous data exchange, such as for serial communication or video streaming. This enables real-time interaction with both -physical and virtual interfaces across the network. +physical and virtual interfaces across the network. Methods marked with the +`@exportstream` decorator create streams for bidirectional communication. + ## Authentication and Security @@ -156,3 +200,86 @@ can be developed for specialized hardware interfaces, emulated environments, or to provide domain-specific abstractions for your use case. Custom drivers follow the same architecture pattern as built-in drivers and can be integrated into the system through the exporter configuration. + +## Example Implementation + +```{testcode} +from sys import modules +from types import SimpleNamespace +from anyio import connect_tcp, sleep +from contextlib import asynccontextmanager +from collections.abc import Generator, AsyncGenerator +from abc import ABCMeta, abstractmethod +from jumpstarter.driver import Driver, export, exportstream +from jumpstarter.client import DriverClient +from jumpstarter.common.utils import serve + +# Define an interface with ABCMeta +class GenericInterface(metaclass=ABCMeta): + @classmethod + def client(cls) -> str: + return "example.GenericClient" + + @abstractmethod + def query(self, param: str) -> str: ... + + @abstractmethod + def get_data(self) -> Generator[dict, None, None]: ... + + @abstractmethod + def create_stream(self): ... + +# Implement the interface with the Driver base class +class GenericDriver(GenericInterface, Driver): + @export + def query(self, param: str) -> str: + # This could be any device-specific command + return f"Response for {param}" + + # driver calls can be either sync or async + @export + async def async_query(self, param: str) -> str: + # Example of an async operation with delay + await sleep(1) + return f"Async response for {param}" + + @export + def get_data(self) -> Generator[dict, None, None]: + # Example of a streaming response - could be sensor data, logs, etc. + for i in range(3): + yield {"type": "data", "value": i, "timestamp": f"2023-04-0{i+1}"} + + # stream constructor has to be an AsyncContextManager + # that yield an anyio.abc.ObjectStream + @exportstream + @asynccontextmanager + async def create_stream(self): + # This could be any stream connection to a device + async with await connect_tcp(remote_host="example.com", remote_port=80) as stream: + yield stream + +class GenericClient(DriverClient): + # client methods are sync + def query(self, param: str) -> str: + return self.call("query", param) + + def async_query(self, param: str) -> str: + # async driver methods can be invoked the same way + return self.call("async_query", param) + + def get_data(self) -> Generator[dict, None, None]: + yield from self.streamingcall("get_data") + + # Streams can be used for bidirectional communication + def with_stream(self, callback): + with self.stream("create_stream") as stream: + callback(stream) + +modules["example"] = SimpleNamespace(GenericClient=GenericClient) + +with serve(GenericDriver()) as client: + assert client.query("test") == "Response for test" + assert client.async_query("async test") == "Async response for async test" + data = list(client.get_data()) + assert len(data) == 3 +``` \ No newline at end of file diff --git a/docs/source/introduction/exporters.md b/docs/source/introduction/exporters.md index 67becd268..039e080f5 100644 --- a/docs/source/introduction/exporters.md +++ b/docs/source/introduction/exporters.md @@ -7,22 +7,22 @@ connected to the target device for client access. ## Hosts -Typically, the host will be a low-cost test system such as a Raspberry Pi or -Mini PC with sufficient interfaces to connect to your hardware. It is also +Typically, the host will be a low-cost test system such as a single board +computer with sufficient interfaces to connect to your hardware. It is also possible to use a local high-power server (or CI runner) as the host device. A host can run multiple Exporter instances simultaneously if it needs to interact with several different devices at the same time. -## Configuration +## Exporter Configuration Exporters use a YAML configuration file to define which Drivers must be loaded and the configuration required. -Here is an example Exporter config file: +Here is an example Exporter config file which would typically be saved at +`/etc/jumpstarter/exporters/demo.yaml`: ```yaml -# /etc/jumpstarter/exporters/myexporter.yaml apiVersion: jumpstarter.dev/v1alpha1 kind: ExporterConfig metadata: @@ -31,7 +31,6 @@ metadata: endpoint: grpc.jumpstarter.example.com:443 token: xxxxx grpcConfig: - # Please refer to the https://grpc.github.io/grpc/core/group__grpc__arg__keys.html documentation grpc.keepalive_time_ms: 20000 export: power: @@ -54,20 +53,25 @@ export: config: hello: "world" reference: - ref: "power" # reference to another driver, this uses the Proxy driver + ref: "power" ``` +Note that the `grpcConfig` section supports all options documented in the [gRPC +argument keys +documentation](https://grpc.github.io/grpc/core/group__grpc__arg__keys.html). + ## Running an Exporter To run an Exporter on a host system, you must have Python {{requires_python}} installed and the driver packages specified in the config installed in your current Python environment. +You can run the exporter in your local terminal with: + ```shell -# Run the exporter myexporter in your local terminal $ jmp run --exporter myexporter ``` Exporters can also be run in a privileged container or as a systemd daemon. It is recommended to run the Exporter service in the background with auto-restart -capabilities in case something goes wrong and it needs to be restarted. +capabilities in case something goes wrong and it needs to be restarted. \ No newline at end of file diff --git a/docs/source/introduction/index.md b/docs/source/introduction/index.md index c15df0beb..46721b970 100644 --- a/docs/source/introduction/index.md +++ b/docs/source/introduction/index.md @@ -1,128 +1,200 @@ # Introduction -Jumpstarter is a free and open source testing tool that enables you to test your -software stack on both real hardware and virtual environments using CI/CD -principles. - -Automated testing with physical hardware (Hardware-in-the-Loop or HiL) and -virtual devices has been established for years in industries such as automotive and -manufacturing. However, these tools are often expensive and inaccessible to -hobbyists and open source projects. - -Jumpstarter provides powerful testing tools that leverage [Cloud -Native](https://www.cncf.io/) principles, modern CI/CD technologies, and open -standards for the next generation of edge devices, whether physical or emulated. - -For a detailed technical overview of the architecture, see the -[Architecture](../architecture.md) documentation. +Jumpstarter is an open source framework that brings enterprise-grade testing +capabilities to everyone. While established industries like automotive and +manufacturing have long used HiL testing, these tools have typically been +expensive proprietary systems. Jumpstarter democratizes this technology through +a free, cloud native approach that works with both physical hardware and virtual +devices. + +At its core, Jumpstarter uses a client/server architecture where a single client +can control multiple devices under test. Its modular design supports both local +development (devices connected directly to your machine) and distributed testing +environments (devices accessed remotely through a central controller). All +communication happens over gRPC, providing a consistent interface regardless of +deployment model. + +Built on Python, Jumpstarter integrates easily with existing development +workflows and runs almost anywhere. It works with common testing tools like +[pytest](https://docs.pytest.org/en/stable/), shell scripts, Makefiles, and +typical CI/CD systems. Beyond testing, it can function as a virtual KVM +(Keyboard, Video, Mouse) switch, enabling remote access to physical devices for +development. ## Core Components -Jumpstarter consists of the following core components: - -- [Clients](./clients.md) - Python library and CLI tools that allow you to - interact with your devices -- [Drivers](./drivers.md) - Modular interfaces that define how to interact with - specific hardware or virtual interfaces -- [Adapters](./adapters.md) - Components that transform driver connections into - different forms or interfaces for specific use cases -- [Exporters](./exporters.md) - Services that expose device interfaces using - drivers -- [Service](./service.md) - Kubernetes-based controller that manages device - access - -### Key Component Relationships - -The relationship between these components is important to understand: - -- **Drivers and Adapters**: Drivers establish and manage the basic connections to hardware or virtual interfaces, while adapters transform these connections into different forms without modifying the underlying driver. This separation creates a flexible architecture where drivers focus on core functionality and adapters enhance usability for specific scenarios. +Jumpstarter architecture is based on the following key components: -- **Exporters and Drivers**: Exporters use drivers to expose device interfaces to clients. The exporter loads driver implementations based on configuration and makes them available over the network. +- Device Under Test (DUT) - Hardware or virtual device being tested +- [Drivers](drivers.md) - Interfaces for DUT communication +- [Adapters](adapters.md) - Convert driver connections into various formats +- [Exporters](exporters.md) - Expose device interfaces over network via gRPC +- [Clients](clients.md) - Libraries and CLI tools for device interaction +- [Service](service.md) - Kubernetes controller for resource management -- **Clients and Exporters**: Clients connect to exporters to access and control the devices. In local mode, this happens directly, while in distributed mode, this is managed by the central controller. +Component interactions include: -Together, these components form a layered architecture that separates concerns and allows for a flexible, extensible testing system. +- **DUT and Drivers** - Drivers provide standardized interfaces to DUT's + hardware connections +- **Drivers and Adapters** - Adapters transform driver connections for + specialized use cases +- **Drivers/Adapters and Exporters** - Exporters manage drivers/adapters and + expose them via gRPC +- **Exporters and Clients** - Clients connect to exporters to control devices +- **Clients/Exporters and Service** - Service manages access control and + resource allocation in distributed mode -## Development Environment - -Since Jumpstarter's core components are written in Python, they can run almost -anywhere. This means that you can set up a test lab with physical hardware or -emulated devices (using tools like QEMU), while still using the same Linux-based -CI systems you currently host in the cloud. - -Jumpstarter integrates seamlessly with the existing ecosystem of Python testing -tools such as [pytest](https://docs.pytest.org/en/stable/). You can also use the -Jumpstarter CLI directly from shell scripts and Makefiles allowing you to write -simple automation scripts easily. - -In addition to testing, Jumpstarter can also act as a KVM (Keyboard, Video, -Mouse) switch - a hardware device that allows multiple computers to share a -single set of input/output devices. Similarly, Jumpstarter enables developers to -remotely access and control both physical and virtual devices for ad-hoc -development whether they are sitting at the next desk or working remotely. +Together, these components form a comprehensive testing framework that bridges +the gap between development and deployment environments. ## Operation Modes -Jumpstarter can be used in either a *local-only* or *distributed* environment -depending on your development needs. - -### Local-Only Mode - -When using Jumpstarter locally, you can develop drivers, write automated tests, -and control your devices directly from your development machine without -additional infrastructure. - -The *local-only mode* is useful when: +Building on these components, Jumpstarter implements two operation modes that +provide flexibility for different scenarios: *local* and *distributed* modes. + +### Local Mode + +In local mode, clients communicate directly with exporters running on the same +machine or through direct network connections. + +```{mermaid} +:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} +flowchart TB + subgraph "Developer Machine" + Client["Client\n(Python Library/CLI)"] + Exporter["Exporter\n(Local Service)"] + end + + subgraph "Target Devices" + DUT["Physical/Virtual\nDevice Under Test"] + Power["Power Interface"] + Serial["Serial Interface"] + Storage["Storage Interface"] + end + + Client <--> |"gRPC via Socket"| Exporter + Exporter --> Power + Exporter --> Serial + Exporter --> Storage + Power --> DUT + Serial --> DUT + Storage --> DUT +``` -- Working with hardware on your desk or virtual devices that you have unlimited - access to -- Developing and testing drivers for new hardware or emulated environments -- Creating initial test automation scripts -- Using QEMU or other virtualization tools to emulate target devices +This mode is ideal for individual developers working directly with accessible +hardware or virtual devices. When no client configuration or environment +variables are present, Jumpstarter runs in local mode and communicates with a +built-in exporter service via a local socket connection, requiring no Kubernetes +or other infrastructure. Developers can work with devices on their desk, develop +drivers, create automation scripts, and test with QEMU or other virtualization +tools. + +```shell +$ jmp shell --exporter my-exporter +$ pytest test_device.py +``` -For details on how this mode works, see the [Running Tests -Locally](../architecture.md#local-mode) section in the architecture -documentation. +The example above shows typical local mode usage: first connecting to an +exporter (which manages the device interfaces) using the `jmp shell` command, +and then running tests against the device with pytest. The `--exporter` flag +specifies which exporter configuration to use, allowing you to easily switch +between different hardware or virtual device setups. ### Distributed Mode -As your project grows, Jumpstarter helps you collaborate across teams, implement -CI/CD pipelines, and automate common tasks such as firmware updates. - -The *distributed mode* leverages [Kubernetes](https://kubernetes.io/) to support -the management of multiple devices (physical or virtual) directly from your -existing cluster. This allows for seamless integration with many existing Cloud -Native technologies such as [Tekton](https://tekton.dev), -[ArgoCD](https://argoproj.github.io/cd/), and -[Prometheus](https://prometheus.io/docs/introduction/overview/). - -The distributed mode is ideal when: - -- Multiple teams need access to shared hardware or virtual device resources -- Continuous integration requires scheduled tests on physical or emulated - devices -- Test environments are distributed across multiple locations -- Devices (physical or virtual) need to be managed remotely - -For technical details on this mode, see the [Running Tests Through a Central -Controller](../architecture.md#distributed-mode) section in the architecture -documentation. - -## Getting Started +Distributed mode enables multiple teams to securely share hardware resources +across a network. It uses a Kubernetes-based controller to coordinate access to +exporters, managing leases that grant exclusive access to DUT resources, while +JWT token-based authentication secures all connections between clients and +exporters. + +```{mermaid} +:config: {"theme":"base","themeVariables":{"primaryColor":"#f8f8f8","primaryTextColor":"#000","primaryBorderColor":"#e5e5e5","lineColor":"#3d94ff","secondaryColor":"#f8f8f8","tertiaryColor":"#fff"}} +flowchart TB + subgraph "Kubernetes Cluster" + Controller["Controller\nResource Management"] + Router["Router\nMessage Routing"] + Auth["Authentication\nJWT Tokens"] + end + + subgraph "Test Runners" + Client1["Client 1\n(CI Pipeline)"] + Client2["Client 2\n(Developer)"] + end + + subgraph "Lab Resources" + Exporter1["Exporter 1\n(Physical Hardware)"] + Exporter2["Exporter 2\n(Virtual Devices)"] + subgraph "Devices" + DUT1["Physical Device 1"] + DUT2["Physical Device 2"] + DUT3["Virtual Device"] + end + end + + Client1 <--> |"JWT Authentication"| Auth + Client2 <--> |"JWT Authentication"| Auth + Exporter1 <--> |"JWT Authentication"| Auth + Exporter2 <--> |"JWT Authentication"| Auth + Auth <--> Controller + + Client1 <--> |"gRPC (Authorized)"| Controller + Client2 <--> |"gRPC (Authorized)"| Controller + Controller <--> Router + Router <--> |"gRPC"| Exporter1 + Router <--> |"gRPC"| Exporter2 + Exporter1 --> DUT1 + Exporter1 --> DUT2 + Exporter2 --> DUT3 +``` -To start using Jumpstarter, check out the following guides: +Distributed mode is ideal for environments where teams need to share hardware +resources, especially in CI/CD pipelines requiring scheduled device testing. It +excels in geographically distributed test environments where devices are spread +across multiple locations, and in any scenario requiring centralized management +of testing resources. All these scenarios require a robust security model to +manage access rights and prevent resource conflicts. + +To address these security needs, the distributed mode implements a comprehensive +authentication system that secures access through: + +- **Client Registration** - Clients register in the Kubernetes cluster with + unique identities +- **Token Issuance** - Controller issues JWT tokens to authenticated clients and + exporters +- **Secure Communication** - All gRPC communication between components uses + token authentication +- **Access Control** - Controller enforces permissions based on token identity: + - Which exporters a client can lease + - What actions a client can perform + - Which driver packages can be loaded + +This security model enables dynamic registration of clients and exporters, +allowing fine-grained access control in multi-user environments. For example, CI +pipelines can be granted access only to specific exporters based on their +credentials, ensuring proper resource isolation in shared testing environments. + +The following example shows how to run tests in distributed mode: + +```shell +$ jmp config client use my-client +$ jmp create lease --selector vendor=acme,model=widget-v2 +$ pytest test_device.py +``` -- [Setup a Local Client](../getting-started/setup-local-exporter.md) - For - local-only development -- [Setup a Remote Client](../getting-started/setup-exporter-client.md) - For - distributed environments +The example above demonstrates the distributed mode workflow: first configuring +the client with connection information for the central controller, then +requesting a lease on an exporter that matches specific criteria (using selector +labels), and finally running tests against the acquired DUT. The lease system +ensures exclusive access to the requested resources for the duration of testing, +preventing conflicts with other users or pipelines in the shared environment. ```{toctree} :maxdepth: 1 :hidden: -clients.md drivers.md adapters.md exporters.md +clients.md service.md -``` +``` \ No newline at end of file diff --git a/docs/source/introduction/service.md b/docs/source/introduction/service.md index d468b7d01..f06588d17 100644 --- a/docs/source/introduction/service.md +++ b/docs/source/introduction/service.md @@ -4,9 +4,9 @@ When building a lab with many devices under test, it quickly becomes difficult to keep track of devices, schedule access for automated tests, and perform routine maintenance such as batch updates. -Jumpstarter provides a [Cloud Native](https://www.cncf.io/) service that can be -installed in any [Kubernetes](https://kubernetes.io/) cluster to manage -connected clients and exporters. +Jumpstarter provides a service that can be installed in any +[Kubernetes](https://kubernetes.io/) cluster to manage connected clients and +exporters. If you're already using a Kubernetes-native CI tool such as [Tekton](https://tekton.dev/), [Jenkins X](https://jenkins-x.io/), @@ -16,9 +16,9 @@ can integrate directly into your existing cloud or on-premises cluster. ## Controller -The core of the Service is the Controller, which manages access to -devices, authenticates clients/exporters, and maintains a set of labels to -easily identify specific devices. +The core of the Service is the Controller, which manages access to devices, +authenticates clients/exporters, and maintains a set of labels to easily +identify specific devices. The Controller is implemented as a Kubernetes [controller](https://github.com/jumpstarter-dev/jumpstarter-controller) using @@ -46,5 +46,5 @@ interfaces via the client. Once a lease is established, all traffic between the client and the exporter flows through a router instance. While there may only be one controller, the -router can be scaled with multiple instances to handle traffic between many clients -and exporters simultaneously. +router can be scaled with multiple instances to handle traffic between many +clients and exporters simultaneously. \ No newline at end of file diff --git a/docs/source/reference/index.md b/docs/source/reference/index.md new file mode 100644 index 000000000..8d3242ea5 --- /dev/null +++ b/docs/source/reference/index.md @@ -0,0 +1,19 @@ +# Reference + +This section provides reference documentation for Jumpstarter. The documentation +covers: + +- [API Pages](man-pages/index.md): Command-line tools and utilities + documentation +- [Packages](package-apis/index.md): API documentation for Jumpstarter packages and + components + +These references are useful for developers working with Jumpstarter. + +```{toctree} +:maxdepth: 1 +:hidden: + +man-pages/index.md +package-apis/index.md +``` \ No newline at end of file diff --git a/docs/source/reference/man-pages/index.md b/docs/source/reference/man-pages/index.md new file mode 100644 index 000000000..b55c61e28 --- /dev/null +++ b/docs/source/reference/man-pages/index.md @@ -0,0 +1,17 @@ +# MAN Pages + +This section provides reference documentation for Jumpstarter's command-line +interfaces. The documentation covers: + +- [`jmp`](jmp.md): Main command-line interface for Jumpstarter +- [`j`](j.md): Shorthand utility for quick interactions with Jumpstarter + +These references support both local and distributed deployment modes of +Jumpstarter. + +```{toctree} +:maxdepth: 1 +:hidden: +jmp.md +j.md +``` \ No newline at end of file diff --git a/docs/source/reference/man-pages/j.md b/docs/source/reference/man-pages/j.md new file mode 100644 index 000000000..67e2574c9 --- /dev/null +++ b/docs/source/reference/man-pages/j.md @@ -0,0 +1,20 @@ +# j + +The `j` command is available within the Jumpstarter shell environment (launched +via `jmp shell`). It provides access to driver CLI interfaces configured in your +exporter. + +Usage: + +```shell +j [OPTIONS] COMMAND [ARGS]... +``` + +The available commands depend on which drivers are loaded in your current +session. When you run the `j` command in the shell: + +- Use `j` alone to see all available driver interfaces +- Access specific drivers with `j ` +- Each driver exposes different commands through this interface + +Available commands vary depending on your configured drivers. \ No newline at end of file diff --git a/docs/source/cli/reference/jmp.md b/docs/source/reference/man-pages/jmp.md similarity index 76% rename from docs/source/cli/reference/jmp.md rename to docs/source/reference/man-pages/jmp.md index 18ad2525a..e06265a4a 100644 --- a/docs/source/cli/reference/jmp.md +++ b/docs/source/reference/man-pages/jmp.md @@ -1,7 +1,5 @@ -# jmp-cli-reference - ```{eval-rst} .. click:: jumpstarter_cli.jmp:jmp :prog: jmp :nested: full -``` +``` \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/can.md b/docs/source/reference/package-apis/drivers/can.md new file mode 120000 index 000000000..e95d1991f --- /dev/null +++ b/docs/source/reference/package-apis/drivers/can.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-can/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/corellium.md b/docs/source/reference/package-apis/drivers/corellium.md new file mode 120000 index 000000000..f0926c8dc --- /dev/null +++ b/docs/source/reference/package-apis/drivers/corellium.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-corellium/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/dbus.yaml b/docs/source/reference/package-apis/drivers/dbus.yaml similarity index 100% rename from docs/source/api-reference/drivers/dbus.yaml rename to docs/source/reference/package-apis/drivers/dbus.yaml diff --git a/docs/source/reference/package-apis/drivers/dutlink.md b/docs/source/reference/package-apis/drivers/dutlink.md new file mode 120000 index 000000000..f3a50c1b4 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/dutlink.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-dutlink/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/flashers.md b/docs/source/reference/package-apis/drivers/flashers.md new file mode 120000 index 000000000..66865df3f --- /dev/null +++ b/docs/source/reference/package-apis/drivers/flashers.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-flashers/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/http.md b/docs/source/reference/package-apis/drivers/http.md new file mode 120000 index 000000000..81e126b1f --- /dev/null +++ b/docs/source/reference/package-apis/drivers/http.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-http/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/index.md b/docs/source/reference/package-apis/drivers/index.md similarity index 71% rename from docs/source/api-reference/drivers/index.md rename to docs/source/reference/package-apis/drivers/index.md index 46241a331..c1411370e 100644 --- a/docs/source/api-reference/drivers/index.md +++ b/docs/source/reference/package-apis/drivers/index.md @@ -1,52 +1,75 @@ # Driver Packages -This section documents the drivers from the Jumpstarter packages directory. Each driver is contained in a separate package in the form of `jumpstarter-driver-{name}` and provides specific functionality for interacting with different hardware components and systems. +This section documents the drivers from the Jumpstarter packages directory. Each +driver is contained in a separate package in the form of +`jumpstarter-driver-{name}` and provides specific functionality for interacting +with different hardware components and systems. ## Types of Drivers -Jumpstarter includes several types of drivers organized by their primary function: +Jumpstarter includes several types of drivers organized by their primary +function: ### System Control Drivers + Drivers that control the power state and basic operation of devices: * **[Power](power.md)** (`jumpstarter-driver-power`) - Power control for devices -* **[Raspberry Pi](raspberrypi.md)** (`jumpstarter-driver-raspberrypi`) - Raspberry Pi hardware control -* **[Yepkit](yepkit.md)** (`jumpstarter-driver-yepkit`) - Yepkit hardware control -* **[DUT Link](dutlink.md)** (`jumpstarter-driver-dutlink`) - [DUT Link Board](https://github.com/jumpstarter-dev/dutlink-board) hardware control +* **[Raspberry Pi](raspberrypi.md)** (`jumpstarter-driver-raspberrypi`) - + Raspberry Pi hardware control +* **[Yepkit](yepkit.md)** (`jumpstarter-driver-yepkit`) - Yepkit hardware + control +* **[DUT Link](dutlink.md)** (`jumpstarter-driver-dutlink`) - [DUT Link + Board](https://github.com/jumpstarter-dev/dutlink-board) hardware control ### Communication Drivers + Drivers that provide various communication interfaces: -* **[CAN](can.md)** (`jumpstarter-driver-can`) - Controller Area Network communication -* **[D-Bus](dbus.md)** (`jumpstarter-driver-dbus`) - D-Bus message system interface +* **[CAN](can.md)** (`jumpstarter-driver-can`) - Controller Area Network + communication * **[HTTP](http.md)** (`jumpstarter-driver-http`) - HTTP communication -* **[Network](network.md)** (`jumpstarter-driver-network`) - Network interfaces and configuration -* **[Proxy](proxy.md)** (`jumpstarter-driver-proxy`) - Network proxy functionality -* **[PySerial](pyserial.md)** (`jumpstarter-driver-pyserial`) - Serial port communication -* **[SNMP](snmp.md)** (`jumpstarter-driver-snmp`) - Simple Network Management Protocol -* **[TFTP](tftp.md)** (`jumpstarter-driver-tftp`) - Trivial File Transfer Protocol +* **[Network](network.md)** (`jumpstarter-driver-network`) - Network interfaces + and configuration +* **[PySerial](pyserial.md)** (`jumpstarter-driver-pyserial`) - Serial port + communication +* **[SNMP](snmp.md)** (`jumpstarter-driver-snmp`) - Simple Network Management + Protocol +* **[TFTP](tftp.md)** (`jumpstarter-driver-tftp`) - Trivial File Transfer + Protocol ### Storage and Data Drivers + Drivers that control storage devices and manage data: -* **[OpenDAL](opendal.md)** (`jumpstarter-driver-opendal`) - Open Data Access Layer -* **[SD Wire](sdwire.md)** (`jumpstarter-driver-sdwire`) - SD card switching utilities +* **[OpenDAL](opendal.md)** (`jumpstarter-driver-opendal`) - Open Data Access + Layer +* **[SD Wire](sdwire.md)** (`jumpstarter-driver-sdwire`) - SD card switching + utilities ### Media Drivers + Drivers that handle media streams: -* **[UStreamer](ustreamer.md)** (`jumpstarter-driver-ustreamer`) - Video streaming functionality +* **[UStreamer](ustreamer.md)** (`jumpstarter-driver-ustreamer`) - Video + streaming functionality ### Debug and Programming Drivers + Drivers for debugging and programming devices: -* **[Flashers](flashers.md)** (`jumpstarter-driver-flashers`) - Flash memory programming tools -* **[Probe-RS](probe-rs.md)** (`jumpstarter-driver-probe-rs`) - Debugging probe support +* **[Flashers](flashers.md)** (`jumpstarter-driver-flashers`) - Flash memory + programming tools +* **[Probe-RS](probe-rs.md)** (`jumpstarter-driver-probe-rs`) - Debugging probe + support * **[QEMU](qemu.md)** (`jumpstarter-driver-qemu`) - QEMU virtualization platform -* **[Corellium](corellium.md)** (`jumpstarter-driver-corellium`) - Corellium virtualization platform -* **[U-Boot](uboot.md)** (`jumpstarter-driver-uboot`) - Universal Bootloader interface +* **[Corellium](corellium.md)** (`jumpstarter-driver-corellium`) - Corellium + virtualization platform +* **[U-Boot](uboot.md)** (`jumpstarter-driver-uboot`) - Universal Bootloader + interface ### Utility Drivers + General-purpose utility drivers: * **[Shell](shell.md)** (`jumpstarter-driver-shell`) - Shell command execution @@ -55,7 +78,6 @@ General-purpose utility drivers: :hidden: can.md corellium.md -dbus.md dutlink.md flashers.md http.md @@ -63,7 +85,6 @@ network.md opendal.md power.md probe-rs.md -proxy.md pyserial.md qemu.md raspberrypi.md diff --git a/docs/source/reference/package-apis/drivers/network.md b/docs/source/reference/package-apis/drivers/network.md new file mode 120000 index 000000000..76a48826d --- /dev/null +++ b/docs/source/reference/package-apis/drivers/network.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-network/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/opendal.md b/docs/source/reference/package-apis/drivers/opendal.md new file mode 120000 index 000000000..5b52160eb --- /dev/null +++ b/docs/source/reference/package-apis/drivers/opendal.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-opendal/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/opendal.yaml b/docs/source/reference/package-apis/drivers/opendal.yaml similarity index 100% rename from docs/source/api-reference/drivers/opendal.yaml rename to docs/source/reference/package-apis/drivers/opendal.yaml diff --git a/docs/source/reference/package-apis/drivers/power.md b/docs/source/reference/package-apis/drivers/power.md new file mode 120000 index 000000000..fcd6d81b3 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/power.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-power/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/probe-rs.md b/docs/source/reference/package-apis/drivers/probe-rs.md new file mode 120000 index 000000000..20ab5affb --- /dev/null +++ b/docs/source/reference/package-apis/drivers/probe-rs.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-probe-rs/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/proxy.yaml b/docs/source/reference/package-apis/drivers/proxy.yaml similarity index 100% rename from docs/source/api-reference/drivers/proxy.yaml rename to docs/source/reference/package-apis/drivers/proxy.yaml diff --git a/docs/source/reference/package-apis/drivers/pyserial.md b/docs/source/reference/package-apis/drivers/pyserial.md new file mode 120000 index 000000000..9846c2565 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/pyserial.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-pyserial/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/qemu.md b/docs/source/reference/package-apis/drivers/qemu.md new file mode 120000 index 000000000..58dfe5652 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/qemu.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-qemu/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/raspberrypi.md b/docs/source/reference/package-apis/drivers/raspberrypi.md new file mode 120000 index 000000000..8ba0a0641 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/raspberrypi.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-raspberrypi/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/sdwire.md b/docs/source/reference/package-apis/drivers/sdwire.md new file mode 120000 index 000000000..70f3a0e9e --- /dev/null +++ b/docs/source/reference/package-apis/drivers/sdwire.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-sdwire/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/sdwire.yaml b/docs/source/reference/package-apis/drivers/sdwire.yaml similarity index 100% rename from docs/source/api-reference/drivers/sdwire.yaml rename to docs/source/reference/package-apis/drivers/sdwire.yaml diff --git a/docs/source/reference/package-apis/drivers/shell.md b/docs/source/reference/package-apis/drivers/shell.md new file mode 120000 index 000000000..be27ac8e3 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/shell.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-shell/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/snmp.md b/docs/source/reference/package-apis/drivers/snmp.md new file mode 120000 index 000000000..7fb9918ce --- /dev/null +++ b/docs/source/reference/package-apis/drivers/snmp.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-snmp/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/tftp.md b/docs/source/reference/package-apis/drivers/tftp.md new file mode 120000 index 000000000..8c6467f63 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/tftp.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-tftp/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/drivers/uboot.md b/docs/source/reference/package-apis/drivers/uboot.md new file mode 120000 index 000000000..bd555bf7a --- /dev/null +++ b/docs/source/reference/package-apis/drivers/uboot.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-uboot/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/uboot.yaml b/docs/source/reference/package-apis/drivers/uboot.yaml similarity index 100% rename from docs/source/api-reference/drivers/uboot.yaml rename to docs/source/reference/package-apis/drivers/uboot.yaml diff --git a/docs/source/reference/package-apis/drivers/ustreamer.md b/docs/source/reference/package-apis/drivers/ustreamer.md new file mode 120000 index 000000000..fd612028a --- /dev/null +++ b/docs/source/reference/package-apis/drivers/ustreamer.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-ustreamer/README.md \ No newline at end of file diff --git a/docs/source/api-reference/drivers/ustreamer.yaml b/docs/source/reference/package-apis/drivers/ustreamer.yaml similarity index 100% rename from docs/source/api-reference/drivers/ustreamer.yaml rename to docs/source/reference/package-apis/drivers/ustreamer.yaml diff --git a/docs/source/reference/package-apis/drivers/yepkit.md b/docs/source/reference/package-apis/drivers/yepkit.md new file mode 120000 index 000000000..57b39d1c0 --- /dev/null +++ b/docs/source/reference/package-apis/drivers/yepkit.md @@ -0,0 +1 @@ +../../../../../packages/jumpstarter-driver-yepkit/README.md \ No newline at end of file diff --git a/docs/source/reference/package-apis/index.md b/docs/source/reference/package-apis/index.md new file mode 100644 index 000000000..eff6f0e6e --- /dev/null +++ b/docs/source/reference/package-apis/index.md @@ -0,0 +1,15 @@ +# Package APIs + +This section provides reference documentation for Jumpstarter's package APIs and +components. The documentation covers: + +- [Drivers](drivers/index.md): APIs for various driver categories + +These references are useful for developers extending Jumpstarter or integrating +with custom hardware. + +```{toctree} +:maxdepth: 1 +:hidden: +drivers/index.md +``` \ No newline at end of file diff --git a/packages/jumpstarter-driver-can/README.md b/packages/jumpstarter-driver-can/README.md index 18e17e240..f3d573dd6 100644 --- a/packages/jumpstarter-driver-can/README.md +++ b/packages/jumpstarter-driver-can/README.md @@ -1,6 +1,7 @@ # CAN driver -`jumpstarter-driver-can` provides functionality for interacting with CAN bus connections. +`jumpstarter-driver-can` provides functionality for interacting with CAN bus +connections. ## Installation diff --git a/packages/jumpstarter-driver-corellium/README.md b/packages/jumpstarter-driver-corellium/README.md index ce746e5b1..1eba074fe 100644 --- a/packages/jumpstarter-driver-corellium/README.md +++ b/packages/jumpstarter-driver-corellium/README.md @@ -1,6 +1,7 @@ # Corellium Driver -`jumpstarter-driver-corellium` provides functionality for interacting with [Corellium](https://corellium.com) virtualization platform. +`jumpstarter-driver-corellium` provides functionality for interacting with +[Corellium](https://corellium.com) virtualization platform. ## Installation diff --git a/packages/jumpstarter-driver-dutlink/README.md b/packages/jumpstarter-driver-dutlink/README.md index f0ca42dea..31eb45b3e 100644 --- a/packages/jumpstarter-driver-dutlink/README.md +++ b/packages/jumpstarter-driver-dutlink/README.md @@ -1,6 +1,7 @@ # DUT Link driver -`jumpstarter-driver-dutlink` provides functionality for interacting with DUT Link devices. +`jumpstarter-driver-dutlink` provides functionality for interacting with DUT +Link devices. ## Installation diff --git a/packages/jumpstarter-driver-flashers/README.md b/packages/jumpstarter-driver-flashers/README.md index 3c4ce0d48..5fee568da 100644 --- a/packages/jumpstarter-driver-flashers/README.md +++ b/packages/jumpstarter-driver-flashers/README.md @@ -40,18 +40,14 @@ export: port: "1" ``` -flasher drivers require four children drivers: | Child Driver | Description | -Auto-created | | ------------ | ---------------------------------------------------------------------------------- -| ------------ | | serial | To communicate with the DUT via serial and -drive the bootloader and busybox shell | No | | power | To -power on and off the DUT | -No | | tftp | To serve binaries via TFTP | Yes | | -http | To serve the images via HTTP | Yes | - -In the above example we provide the serial and power children by -[reference](./proxy.md), so those remain also available on the root of the -exporter. +flasher drivers require four children drivers: + +| Child Driver | Description | Auto-created | +| ------------ | --------------------------------------------------------------------------------- | ------------ | +| serial | To communicate with the DUT via serial and drive the bootloader and busybox shell | No | +| power | To power on and off the DUT | No | +| tftp | To serve binaries via TFTP | Yes | +| http | To serve the images via HTTP | Yes | The power driver is used to control power cycling of the DUT, and the serial interface is used to communicate with the DUT bootloader via serial. TFTP and @@ -253,7 +249,8 @@ with flasherclient.bootloader_shell() as serial: print(serial.before) ``` -# oci-bundles +## oci-bundles + The flasher drivers require some artifacts and basic information about the target device to operate. To make this easy to distribute and use, we use OCI bundles to package the artifacts and metadata. @@ -265,10 +262,12 @@ artifacts and metadata. It is a container that contains the following: ## The format of the manifest is as follows: -```{literalinclude} ../../../../packages/jumpstarter-driver-flashers/oci_bundles/test/manifest.yaml +```{literalinclude} ../../../../../packages/jumpstarter-driver-flashers/oci_bundles/test/manifest.yaml :language: yaml ``` + ## Table with the spec fields of the manifest: + | Field | Description | Default | | -------------------- | -------------------------------------------------------------------------- | ------- | | `manufacturer` | Name of the device manufacturer | | @@ -282,20 +281,19 @@ artifacts and metadata. It is a container that contains the following: | `login.password` | Password for login, leave empty if not needed | | | `login.prompt` | Shell prompt after successful login | # | | `preflash_commands` | List of commands to run before flashing, useful to clear boot entries, etc | | -| `kernel.file` | Path to kernel image within bundle | +| `kernel.file` | Path to kernel image within bundle | | | `kernel.address` | Memory address to load kernel to | | | `initram.file` | Path to initramfs within bundle (if any) | | | `initram.address` | Memory address to load initramfs to (if any) | | | `dtb.default` | Default DTB variant to use | | | `dtb.address` | Memory address to load DTB to | | -| `dtb.variants` | Map of DTB variant names to files | - +| `dtb.variants` | Map of DTB variant names to files | | ## Examples An example bundle for the TI J784S4XEVM looks like this: -```{literalinclude} ../../../../packages/jumpstarter-driver-flashers/oci_bundles/ti_j784s4xevm/manifest.yaml +```{literalinclude} ../../../../../packages/jumpstarter-driver-flashers/oci_bundles/ti_j784s4xevm/manifest.yaml :language: yaml ``` diff --git a/packages/jumpstarter-driver-network/README.md b/packages/jumpstarter-driver-network/README.md index 5c08df208..e072d8f64 100644 --- a/packages/jumpstarter-driver-network/README.md +++ b/packages/jumpstarter-driver-network/README.md @@ -1,6 +1,7 @@ # Network drivers -`jumpstarter-driver-network` provides functionality for interacting with network servers and connections. +`jumpstarter-driver-network` provides functionality for interacting with network +servers and connections. ## Installation diff --git a/packages/jumpstarter-driver-opendal/README.md b/packages/jumpstarter-driver-opendal/README.md index d13d497f2..15b60edeb 100644 --- a/packages/jumpstarter-driver-opendal/README.md +++ b/packages/jumpstarter-driver-opendal/README.md @@ -1,6 +1,7 @@ # OpenDAL driver -`jumpstarter-driver-opendal` provides functionality for interacting with storages attached to the exporter. +`jumpstarter-driver-opendal` provides functionality for interacting with +storages attached to the exporter. ## Installation @@ -33,7 +34,7 @@ from jumpstarter.config.exporter import ExporterConfigV1Alpha1DriverInstance from jumpstarter.common.utils import serve instance = serve( - ExporterConfigV1Alpha1DriverInstance.from_path("source/api-reference/drivers/opendal.yaml" + ExporterConfigV1Alpha1DriverInstance.from_path("source/reference/package-apis/drivers/opendal.yaml" ).instantiate()) opendal = instance.__enter__() diff --git a/packages/jumpstarter-driver-power/README.md b/packages/jumpstarter-driver-power/README.md index f599e3555..54f439008 100644 --- a/packages/jumpstarter-driver-power/README.md +++ b/packages/jumpstarter-driver-power/README.md @@ -1,6 +1,7 @@ # Power driver -`jumpstarter-driver-power` provides functionality for interacting with power control devices. +`jumpstarter-driver-power` provides functionality for interacting with power +control devices. ## Installation diff --git a/packages/jumpstarter-driver-probe-rs/README.md b/packages/jumpstarter-driver-probe-rs/README.md index 567a03a72..2e64012a4 100644 --- a/packages/jumpstarter-driver-probe-rs/README.md +++ b/packages/jumpstarter-driver-probe-rs/README.md @@ -1,6 +1,7 @@ # probe-rs driver -`jumpstarter-driver-probe-rs` provides functionality for remote debugging and flashing of embedded devices using the [probe-rs](https://probe.rs) tools. +`jumpstarter-driver-probe-rs` provides functionality for remote debugging and +flashing of embedded devices using the [probe-rs](https://probe.rs) tools. ## Installation @@ -43,7 +44,8 @@ export: ### CLI -The probe driver client comes with a CLI tool that can be used to interact with the target device. +The probe driver client comes with a CLI tool that can be used to interact with +the target device. ``` jumpstarter โšก local โžค j probe Usage: j probe [OPTIONS] COMMAND [ARGS]... diff --git a/packages/jumpstarter-driver-pyserial/README.md b/packages/jumpstarter-driver-pyserial/README.md index 25b30a44d..19dd7b896 100644 --- a/packages/jumpstarter-driver-pyserial/README.md +++ b/packages/jumpstarter-driver-pyserial/README.md @@ -1,6 +1,7 @@ # PySerial driver -`jumpstarter-driver-pyserial` provides functionality for serial port communication. +`jumpstarter-driver-pyserial` provides functionality for serial port +communication. ## Installation diff --git a/packages/jumpstarter-driver-qemu/README.md b/packages/jumpstarter-driver-qemu/README.md index d9690e902..b2ba79300 100644 --- a/packages/jumpstarter-driver-qemu/README.md +++ b/packages/jumpstarter-driver-qemu/README.md @@ -1,6 +1,7 @@ # QEMU driver -`jumpstarter-driver-qemu` provides functionality for interacting with QEMU virtualization platform. +`jumpstarter-driver-qemu` provides functionality for interacting with QEMU +virtualization platform. ## Installation diff --git a/packages/jumpstarter-driver-raspberrypi/README.md b/packages/jumpstarter-driver-raspberrypi/README.md index 7edeb906c..b6a26e8cd 100644 --- a/packages/jumpstarter-driver-raspberrypi/README.md +++ b/packages/jumpstarter-driver-raspberrypi/README.md @@ -1,6 +1,7 @@ # Raspberry Pi driver -`jumpstarter-driver-raspberrypi` provides functionality for interacting with Raspberry Pi devices. +`jumpstarter-driver-raspberrypi` provides functionality for interacting with +Raspberry Pi devices. ## Installation diff --git a/packages/jumpstarter-driver-sdwire/README.md b/packages/jumpstarter-driver-sdwire/README.md index 6b42bd3b8..451a10447 100644 --- a/packages/jumpstarter-driver-sdwire/README.md +++ b/packages/jumpstarter-driver-sdwire/README.md @@ -1,6 +1,8 @@ # SDWire driver -`jumpstarter-driver-sdwire` provides functionality for using the SDWire storage multiplexer. This device multiplexes an SD card between the DUT and the exporter host. +`jumpstarter-driver-sdwire` provides functionality for using the SDWire storage +multiplexer. This device multiplexes an SD card between the DUT and the exporter +host. ## Installation @@ -27,7 +29,8 @@ FileNotFoundError: failed to find sd-wire device ## API Reference -The SDWire driver implements the `StorageMuxClient` class, which is a generic storage class. +The SDWire driver implements the `StorageMuxClient` class, which is a generic +storage class. ```{eval-rst} .. autoclass:: jumpstarter_driver_opendal.client.StorageMuxClient() diff --git a/packages/jumpstarter-driver-shell/README.md b/packages/jumpstarter-driver-shell/README.md index 68cd209f7..7691d4b81 100644 --- a/packages/jumpstarter-driver-shell/README.md +++ b/packages/jumpstarter-driver-shell/README.md @@ -35,7 +35,8 @@ export: ## API Reference -Assuming the exporter driver is configured as in the example above, the client methods will be generated dynamically, and they will be available as follows: +Assuming the exporter driver is configured as in the example above, the client +methods will be generated dynamically, and they will be available as follows: ```{eval-rst} .. autoclass:: jumpstarter_driver_shell.client.ShellClient diff --git a/packages/jumpstarter-driver-snmp/README.md b/packages/jumpstarter-driver-snmp/README.md index f42177fea..86916fa66 100644 --- a/packages/jumpstarter-driver-snmp/README.md +++ b/packages/jumpstarter-driver-snmp/README.md @@ -1,6 +1,7 @@ # SNMP driver -`jumpstarter-driver-snmp` provides functionality for controlling power via SNMP-enabled PDUs (Power Distribution Units). +`jumpstarter-driver-snmp` provides functionality for controlling power via +SNMP-enabled PDUs (Power Distribution Units). ## Installation diff --git a/packages/jumpstarter-driver-tftp/README.md b/packages/jumpstarter-driver-tftp/README.md index c35043e03..83fc9a6d4 100644 --- a/packages/jumpstarter-driver-tftp/README.md +++ b/packages/jumpstarter-driver-tftp/README.md @@ -1,6 +1,7 @@ # TFTP driver -`jumpstarter-driver-tftp` provides functionality for a read-only TFTP server that can be used to serve files. +`jumpstarter-driver-tftp` provides functionality for a read-only TFTP server +that can be used to serve files. ## Installation diff --git a/packages/jumpstarter-driver-uboot/README.md b/packages/jumpstarter-driver-uboot/README.md index a24dff85b..d0d95dcea 100644 --- a/packages/jumpstarter-driver-uboot/README.md +++ b/packages/jumpstarter-driver-uboot/README.md @@ -1,6 +1,8 @@ # U-Boot driver -`jumpstarter-driver-uboot` provides functionality for interacting with the U-Boot bootloader. This driver does not interact with the DUT directly, instead it should be configured with backing power and serial drivers. +`jumpstarter-driver-uboot` provides functionality for interacting with the +U-Boot bootloader. This driver does not interact with the DUT directly, instead +it should be configured with backing power and serial drivers. ## Installation @@ -19,7 +21,7 @@ Example configuration: ```{doctest} :hide: >>> from jumpstarter.config.exporter import ExporterConfigV1Alpha1DriverInstance ->>> ExporterConfigV1Alpha1DriverInstance.from_path("source/api-reference/drivers/uboot.yaml").instantiate() +>>> ExporterConfigV1Alpha1DriverInstance.from_path("source/reference/package-apis/drivers/uboot.yaml").instantiate() UbootConsole(...) ``` diff --git a/packages/jumpstarter-driver-ustreamer/README.md b/packages/jumpstarter-driver-ustreamer/README.md index d64b67aaa..f2937ffa3 100644 --- a/packages/jumpstarter-driver-ustreamer/README.md +++ b/packages/jumpstarter-driver-ustreamer/README.md @@ -1,6 +1,8 @@ # Ustreamer driver -`jumpstarter-driver-ustreamer` provides functionality for using the ustreamer video streaming server driven by the jumpstarter exporter. This driver takes a video device and exposes both snapshot and streaming interfaces. +`jumpstarter-driver-ustreamer` provides functionality for using the ustreamer +video streaming server driven by the jumpstarter exporter. This driver takes a +video device and exposes both snapshot and streaming interfaces. ## Installation @@ -19,7 +21,7 @@ Example configuration: ```{doctest} :hide: >>> from jumpstarter.config.exporter import ExporterConfigV1Alpha1DriverInstance ->>> ExporterConfigV1Alpha1DriverInstance.from_path("source/api-reference/drivers/ustreamer.yaml").instantiate() +>>> ExporterConfigV1Alpha1DriverInstance.from_path("source/reference/package-apis/drivers/ustreamer.yaml").instantiate() Traceback (most recent call last): ... io.UnsupportedOperation: fileno diff --git a/packages/jumpstarter-driver-yepkit/README.md b/packages/jumpstarter-driver-yepkit/README.md index 218cd672e..acecadd7a 100644 --- a/packages/jumpstarter-driver-yepkit/README.md +++ b/packages/jumpstarter-driver-yepkit/README.md @@ -1,6 +1,7 @@ # Yepkit driver -`jumpstarter-driver-yepkit` provides functionality for interacting with Yepkit products. +`jumpstarter-driver-yepkit` provides functionality for interacting with Yepkit +products. ## Installation