From a518478e84937c70105bd4181c5c2e387e3cd301 Mon Sep 17 00:00:00 2001 From: MikiEEE <35975462+MikiEEE@users.noreply.github.com> Date: Fri, 28 Aug 2026 23:27:23 -0500 Subject: [PATCH] docs: add multipage guide and demo walkthroughs --- README.md | 550 +++------------------------ demo.py | 3 + demos/adapters_asyncio_demo.py | 6 + demos/adapters_demo.py | 6 + demos/adapters_sqlite_demo.py | 8 +- demos/common.py | 12 + demos/esp32_demo.py | 4 + demos/http_demo.py | 4 + demos/micropython_autodetect_demo.py | 3 + demos/mqtt_demo.py | 4 + demos/pico_w_demo.py | 3 + demos/redis_demo.py | 3 + demos/runtime_demo.py | 17 + demos/shell_demo.py | 8 + demos/unix_demo.py | 4 + demos/web_app_demo.py | 16 + guide/README.md | 34 ++ guide/api-reference.md | 122 ++++++ guide/configuration.md | 67 ++++ guide/contributing.md | 44 +++ guide/demos.md | 45 +++ guide/error-handling.md | 79 ++++ guide/execution-adapters.md | 99 +++++ guide/installation.md | 137 +++++++ guide/kernels-and-micropython.md | 61 +++ guide/networking-clients.md | 48 +++ guide/quick-start.md | 57 +++ guide/runtime-concepts.md | 68 ++++ guide/shell-and-server.md | 117 ++++++ guide/task-lifecycle.md | 152 ++++++++ guide/troubleshooting.md | 88 +++++ 31 files changed, 1364 insertions(+), 505 deletions(-) create mode 100644 guide/README.md create mode 100644 guide/api-reference.md create mode 100644 guide/configuration.md create mode 100644 guide/contributing.md create mode 100644 guide/demos.md create mode 100644 guide/error-handling.md create mode 100644 guide/execution-adapters.md create mode 100644 guide/installation.md create mode 100644 guide/kernels-and-micropython.md create mode 100644 guide/networking-clients.md create mode 100644 guide/quick-start.md create mode 100644 guide/runtime-concepts.md create mode 100644 guide/shell-and-server.md create mode 100644 guide/task-lifecycle.md create mode 100644 guide/troubleshooting.md diff --git a/README.md b/README.md index 125b435..1e41239 100644 --- a/README.md +++ b/README.md @@ -1,135 +1,42 @@ # smallOS `smallOS` is a lightweight cooperative runtime for priority-oriented task -management. +management. It keeps the familiar `async` / `await` syntax while letting +smallOS—not `asyncio`—own scheduling policy. -It is designed around three ideas: -- write tasks with modern `async` / `await` syntax -- keep scheduling policy owned by `smallOS`, not `asyncio` -- stay portable enough to run on desktop Python today and MicroPython boards - later - -## Status - -The project is currently experimental but usable. The runtime core supports: -- priority-based cooperative scheduling -- task spawning, `join`, and `join_all` -- signal-based wakeups -- time-based sleeping -- readiness-based socket/I/O waiting -- generic TCP/TLS kernel hooks for higher-level protocols -- dependency-free thread and asyncio execution adapters for user libraries -- smallOS-native HTTP, Redis, MQTT, SSE, and WebSocket helper clients +The project is experimental but usable. It currently provides priority-based +scheduling, task joins and signals, time and socket waits, desktop and +MicroPython kernels, execution adapters for blocking or asyncio-owned code, +and dependency-free HTTP, Redis, MQTT, SSE, and WebSocket clients. ## Why smallOS? -Python's `asyncio` gives great syntax, but it also brings its own scheduler and -event-loop policy. `smallOS` keeps the syntax while swapping in a custom -runtime, so tasks can be scheduled with project-specific priority rules and a -smaller portability surface. +smallOS is designed for projects that need a small, explicit runtime surface: -That makes it a good fit for: -- robotics or device-control projects -- embedded experiments on MicroPython boards -- custom runtimes where task priority matters -- learning how coroutine scheduling works under the hood +- robotics and device-control experiments +- MicroPython and embedded exploration +- applications where task priority matters +- learning how coroutine schedulers work -## Project Layout +Arbitrary asyncio libraries are not drop-in compatible because smallOS steps +its own coroutines. When an existing library owns blocking or asyncio work, +the runtime provides opt-in execution adapters. -- [SmallPackage/SmallOS.py](SmallPackage/SmallOS.py): - cooperative scheduler -- [SmallPackage/SmallTask.py](SmallPackage/SmallTask.py): - task lifecycle, coroutine stepping, join bookkeeping -- [SmallPackage/Kernel.py](SmallPackage/Kernel.py): - desktop and MicroPython kernel abstractions -- [SmallPackage/SmallIO.py](SmallPackage/SmallIO.py): - buffered app/shell output routing and terminal-mode helpers -- [SmallPackage/clients](SmallPackage/clients): - protocol client package for cooperative network integrations -- [SmallPackage/adapters](SmallPackage/adapters): - dependency-free escape hatches for blocking and asyncio-owned user code -- [SmallPackage/clients/README.md](SmallPackage/clients/README.md): - detailed client-specific guide and API notes -- [SmallPackage/clients/SmallHTTP.py](SmallPackage/clients/SmallHTTP.py): - dependency-free HTTP and SSE clients for smallOS tasks -- [SmallPackage/clients/SmallStream.py](SmallPackage/clients/SmallStream.py): - cooperative socket stream helper for protocol clients -- [SmallPackage/clients/SmallRedis.py](SmallPackage/clients/SmallRedis.py): - dependency-free Redis client for smallOS tasks -- [SmallPackage/clients/SmallMQTT.py](SmallPackage/clients/SmallMQTT.py): - dependency-free MQTT client for smallOS tasks -- [SmallPackage/clients/SmallWebSocket.py](SmallPackage/clients/SmallWebSocket.py): - dependency-free WebSocket client for bidirectional messaging -- [SmallPackage/SmallConfig.py](SmallPackage/SmallConfig.py): - runtime configuration loader/container -- [smallos.config.json](smallos.config.json): - repo-level runtime defaults -- [SmallPackage/shells.py](SmallPackage/shells.py): - command shell helpers for runtime inspection and demos -- [demos](demos): - desktop and board-specific demo entry points -- [tests](tests): - unit tests for scheduler, kernel, config, and supporting structures +## Quick start -## Installation - -Desktop development: +smallOS supports CPython 3.10 and newer. ```bash python3.10 -m venv .venv source .venv/bin/activate python -m pip install -e ".[dev]" +python demos/unix_demo.py ``` -smallOS supports CPython 3.10 and newer. Python 3.6 through 3.9 are no -longer supported. MicroPython compatibility is maintained separately because -its language and standard-library support do not map directly to a CPython -release number; checker-only imports are kept off embedded runtime paths. - -Run the test suite: - -```bash -python3 -m unittest discover -s tests -v -``` - -Run the tests with the same branch-coverage gate used by CI: - -```bash -coverage run -m unittest discover -s tests -v -coverage report -``` - -Run the static type checker: - -```bash -pyright -``` - -Build the wheel and source distribution: - -```bash -python -m build -``` - -The GitHub Actions pipeline runs four gates: Pyright, unit tests across Python -3.10–3.13, branch coverage with a 60% floor, and distribution verification. -Packaging runs only after the earlier gates pass, installs the built wheel, -and smoke-tests it outside the source checkout. - -The package ships a `py.typed` marker. Type coverage is being tightened by -subsystem: configuration, awaitables, task lifecycle, scheduling, signals, -platform kernels, and core utilities form the current checked boundary, while -protocol clients, shells, and demos remain on the incremental typing backlog. - -## Quick Start - -Minimal desktop runtime: +Minimal runtime: ```python -from SmallPackage.Kernel import Unix -from SmallPackage.SmallConfig import SmallOSConfig -from SmallPackage.SmallOS import SmallOS -from SmallPackage.SmallTask import SmallTask +from SmallPackage import SmallOS, SmallOSConfig, SmallTask, Unix async def hello(task): @@ -140,402 +47,37 @@ async def hello(task): config = SmallOSConfig.from_json_file("smallos.config.json") runtime = SmallOS(config=config).setKernel(Unix()) -runtime.setErrorHandler( - lambda event: print( - "[smallOS] task failure in {} (PID {}): {}".format( - event["task_name"] or "unnamed task", - event["task_id"], - event["exception_repr"], - ) - ) -) runtime.fork([SmallTask(2, hello, name="hello")]) runtime.startOS() ``` -## Runtime Error Handling - -`smallOS` now supports a runtime-level error observer through -`runtime.setErrorHandler(handler, include_cancelled=False)`. - -Use it when you want: -- readable debug output for uncaught task failures -- lightweight cleanup or bookkeeping at the runtime boundary -- a single place to surface task errors without crashing the scheduler - -The handler is synchronous and receives a failure-event dictionary after the -task has been finalized. Current event fields include: -- `task_id` -- `task_name` -- `parent_id` -- `exception` -- `exception_type` -- `exception_repr` -- `is_cancelled` -- `blocked_reason` -- `waiting_signal` -- `io_wait_mode` -- `join_target_id` -- `join_pending_ids` -- `adapter_name` -- `adapter_job_id` -- `traceback_text` - -By default, `TaskCancelledError` does not trigger the handler. Pass -`include_cancelled=True` if you want cancellation events too. - -Example: - -```python -from SmallPackage.Kernel import Unix -from SmallPackage.SmallOS import SmallOS - - -def log_runtime_error(event): - print( - "[smallOS] task failure in {} (PID {}): {}".format( - event["task_name"] or "unnamed task", - event["task_id"], - event["exception_repr"], - ) - ) - if event["traceback_text"]: - print(event["traceback_text"], end="") - - -runtime = SmallOS().setKernel(Unix()) -runtime.setErrorHandler(log_runtime_error) -``` - -### Closed or Invalid File Descriptors - -Closed or invalid file descriptors used in `wait_readable(...)` or -`wait_writable(...)` no longer crash the whole scheduler through the platform -poll/select layer. - -Instead: -- the kernel validates the watched object before polling -- the waiting task receives a normal exception such as `ValueError` -- the runtime finalizes that task cleanly -- your runtime error handler can log or clean up the failure gracefully - -If you do not install an error handler, the task still fails cleanly and the -runtime keeps its internal state consistent, but adding `setErrorHandler(...)` -is the recommended way to make these failures visible in applications. - -### Waking a Blocked Scheduler from Another Thread - -Kernels may provide an opaque wakeup channel for code that must request work -such as server shutdown while the scheduler is blocked in I/O readiness: - -```python -kernel = Unix() -if not kernel.supports_wakeup_channel(): - raise RuntimeError("cross-thread scheduler wakeup is unavailable") - -wakeup = kernel.create_wakeup_channel() - -async def watch_shutdown(task): - await task.wait_readable(wakeup.wait_object) - wakeup.drain() - # Apply the application-owned shutdown request on the scheduler thread. -``` - -Call `wakeup.notify()` from the external thread. Notifications are nonblocking -and coalesce until the scheduler calls `drain()`. The owner must call `close()` -after its scheduler wait has been detached; repeated close, notify, and drain -calls during teardown are safe. - -`Unix` supports this contract when its socket module provides a callable -`socketpair()`. Generic `MicroPythonKernel` deliberately reports it unsupported: -a polling backend or socket-pair-shaped attribute alone does not establish safe -cross-thread behavior on a constrained port. TCP serving and shutdown initiated -by a task already running on the scheduler do not require this capability. - -## Configuration - -The runtime now uses a first-class config object backed by -[smallos.config.json](smallos.config.json). - -Current config fields: -- `task_capacity`: maximum tracked tasks / PID slots -- `priority_levels`: number of ready-queue categories -- `io_buffer_length`: buffered app output length when the terminal view is hidden -- `eternal_watchers`: keep the runtime alive when only watcher tasks remain -- `client_defaults`: shared defaults for cooperative clients and streams - -Example: - -```json -{ - "task_capacity": 1024, - "priority_levels": 10, - "io_buffer_length": 1024, - "eternal_watchers": false, - "client_defaults": { - "stream": { - "max_buffer_size": 16777216 - }, - "http": { - "max_response_size": 16777216 - }, - "redis": { - "max_response_size": 16777216, - "max_nesting_depth": 32 - }, - "mqtt": { - "keepalive": 60, - "max_packet_size": 262144, - "max_queued_messages": 1024 - } - } -} -``` - -The config loader also accepts the aliases `oslist_length` and -`num_categories` so older notes and experiments can map cleanly onto the -current runtime. - -Client constructors still accept explicit overrides, but when you create them -inside a task they now inherit these defaults from `task.OS.config` unless you -pass a value directly. - -## Kernels and Board Profiles - -Desktop kernel: -- `Unix` - -MicroPython kernels: -- `MicroPythonKernel` -- `ESP32` -- `PicoW` / `RaspberryPiPicoW` -- `ESP8266` compatibility profile - -You can either pick a board profile explicitly or let the runtime choose a -built-in profile from the firmware machine string: - -```python -from SmallPackage.Kernel import ESP32, PicoW, build_micropython_kernel - -kernel = ESP32(hostname="smallos-esp32") -kernel = PicoW(country="US", hostname="smallos-pico") -kernel = build_micropython_kernel() -``` - -## Demos - -The new demos live in [demos](demos): -- [demos/unix_demo.py](demos/unix_demo.py): - desktop scheduler demo -- [demos/esp32_demo.py](demos/esp32_demo.py): - ESP32-oriented startup and optional Wi-Fi bring-up example -- [demos/pico_w_demo.py](demos/pico_w_demo.py): - Pico W oriented startup and Wi-Fi configuration example -- [demos/micropython_autodetect_demo.py](demos/micropython_autodetect_demo.py): - automatic MicroPython kernel selection -- [demos/runtime_demo.py](demos/runtime_demo.py): - migrated home for the original root-level runtime showcase -- [demos/shell_demo.py](demos/shell_demo.py): - scripted shell session running alongside other cooperative tasks -- [demos/redis_demo.py](demos/redis_demo.py): - Redis example built on the native cooperative client -- [demos/http_demo.py](demos/http_demo.py): - HTTP example built on the native cooperative client -- [demos/web_app_demo.py](demos/web_app_demo.py): - cooperative single-thread web app demo with HTTP routes, live browser UI, and shell-driven server shutdown -- [demos/mqtt_demo.py](demos/mqtt_demo.py): - MQTT example built on the native cooperative client -- [demos/adapters_demo.py](demos/adapters_demo.py): - thread and asyncio escape hatches running beside a regular SmallOS task -- [demos/adapters_sqlite_demo.py](demos/adapters_sqlite_demo.py): - a user-owned `sqlite3` connection kept on one thread-adapter worker -- [demos/adapters_asyncio_demo.py](demos/adapters_asyncio_demo.py): - a persistent asyncio queue and background task reused across adapter calls - -All of the shared demo entry points now install a default runtime error handler -through [demos/common.py](demos/common.py). That means network failures, -invalid I/O wait objects, and other uncaught task exceptions are reported as -readable task-failure diagnostics instead of looking like abrupt scheduler -crashes or silent exits. - -The original root demo remains available in -[demo.py](demo.py) as a compatibility wrapper around -[demos/runtime_demo.py](demos/runtime_demo.py). - -## Runtime Model - -`smallOS` is intentionally small in scope: -- tasks are `async def` coroutines wrapped in `SmallTask` -- task code awaits smallOS-owned awaitables such as `task.sleep(...)`, - `task.wait_signal(...)`, `task.wait_readable(...)`, and `task.join(...)` -- the scheduler steps coroutines directly and decides when each task becomes - runnable again -- kernels provide timing, output, and readiness-based transport primitives - -This means arbitrary `asyncio` libraries are not drop-in compatible with the -runtime, but it also means scheduling policy and portability stay under your -control. - -## Execution Adapters - -Execution adapters let a SmallOS task yield while user-supplied code runs under -a different execution model. They use only the Python standard library and do -not install, import, configure, or wrap database drivers, ORMs, SDKs, or other -third-party packages. - -Use `ThreadAdapter` for a synchronous blocking callable: - -```python -from SmallPackage.adapters.threads import ThreadAdapter - - -def load_record(user_library, settings, record_id): - connection = user_library.connect(**settings) - try: - return connection.load(record_id) - finally: - connection.close() - - -with ThreadAdapter(max_workers=4, max_pending=64) as blocking: - async def load(task): - return await blocking.call( - load_record, - user_selected_library, - connection_settings, - 42, - ) - - runtime.fork(SmallTask(2, load, name="load")) - runtime.start() -``` - -Use `AsyncioAdapter` for an async callable that must run on asyncio. The -adapter owns one persistent event loop in a dedicated thread, allowing -loop-affine clients to be reused when all their operations are routed through -the same adapter: - -```python -from SmallPackage.adapters.asyncio_loop import AsyncioAdapter - - -async def fetch_record(user_library, settings, record_id): - async with user_library.Client(**settings) as client: - return await client.fetch(record_id) - - -with AsyncioAdapter(max_pending=64) as foreign_async: - async def load(task): - return await foreign_async.call( - fetch_record, - user_selected_async_library, - connection_settings, - 42, - ) - - runtime.fork(SmallTask(2, load, name="load")) - runtime.start() -``` - -Important behavior: - -- adapters bind to the first SmallOS runtime that uses them; -- adapter shutdown is explicit, so a context manager should wrap - `runtime.start()`; -- `max_pending` rejects excess work with `AdapterCapacityError` instead of - blocking the scheduler; -- cancelling a SmallTask can cancel queued thread work, but cannot forcibly - stop a running Python thread; -- asyncio cancellation is requested on the adapter loop, but a user library - may delay or suppress it; -- pass an async callable to `AsyncioAdapter.call()`, not a `Task` or `Future` - already owned by another loop; -- inspect `AsyncioAdapter.shutdown_error` after shutdown when application - diagnostics need to detect an unexpected loop stop or library teardown - failure; normal shutdown leaves it as `None`; -- use `ThreadAdapter(max_workers=1)` when user resources require a serialized, - thread-affine execution lane; create, use, and close those resources through - calls on that same adapter rather than creating them on the SmallOS thread. - -### Standard-library examples - -All adapter demos run without installing anything beyond SmallOS: - -```bash -python3 demos/adapters_demo.py -python3 demos/adapters_sqlite_demo.py -python3 demos/adapters_asyncio_demo.py -``` - -- [demos/adapters_demo.py](demos/adapters_demo.py) runs both adapters beside an - ordinary cooperative SmallOS task. -- [demos/adapters_sqlite_demo.py](demos/adapters_sqlite_demo.py) creates, uses, - and closes an in-memory `sqlite3` connection through - `ThreadAdapter(max_workers=1)`. This is the ownership pattern to adapt for a - thread-affine PostgreSQL driver or ORM session supplied by the user. -- [demos/adapters_asyncio_demo.py](demos/adapters_asyncio_demo.py) creates an - `asyncio.Queue`, `Future` objects, and a background task, performs multiple - operations, and cleans them up on the same persistent adapter event loop. - -## Running on MicroPython - -For MicroPython targets, the intended flow is: -1. choose `ESP32`, `PicoW`, or `build_micropython_kernel()` -2. optionally connect Wi-Fi through the kernel helper -3. build `SmallOS(config=...)` -4. fork tasks and start the runtime - -The kernel layer is deliberately generic. Protocol clients such as HTTPS, -Redis, MQTT, RabbitMQ/AMQP, and Kafka should be built on top of the shared -TCP/TLS socket surface rather than requiring protocol-specific kernel methods. - -Passive TCP consumers use the kernel boundary as well: check -`supports_tcp_server()` before resolving or opening anything, pass the opaque -record returned by `resolve_passive_address()` unchanged to both `socket_open()` -and `socket_bind()`, then use the kernel's listen, accept, address-inspection, -and close operations. Address reuse has its own capability check because some -MicroPython ports support listeners without exposing `SO_REUSEADDR` constants. -The web app demo shows the complete setup and rollback pattern without importing -platform socket APIs. - -## Clients - -The current setup now includes first-party smallOS-native helpers for HTTP, -Redis, and MQTT, so users can stay inside the smallOS scheduler instead of -dropping down to raw sockets or depending on `asyncio`-owned clients. - -Available helpers: -- `SmallHTTPClient` -- `SmallRedisClient` -- `SmallMQTTClient` - -Current scope: -- HTTP: request/response helper with query params, JSON bodies, TLS, and - chunked/content-length response parsing -- Redis: RESP command execution plus helpers like `ping`, `get`, `set`, - `delete`, `publish`, and `subscribe` -- MQTT: MQTT 3.1.1 connect/disconnect, publish at QoS 0/1/2, subscribe at - QoS 0/1/2, and inbound message receive with PUBACK/PUBREC/PUBREL/PUBCOMP - handling as required by the protocol -- Both clients: optional username/password auth and TLS transport setup, with - Unix support for custom CA and client certificate paths through - `tls_ca_file`, `tls_cert_file`, and `tls_key_file` - -For detailed examples, constructor options, response helpers, and transport -notes, see -[SmallPackage/clients/README.md](SmallPackage/clients/README.md). - -## Contributing - -Contributions, issues, experiments, and board-port notes are welcome. Good -areas for contribution include: -- new MicroPython port validation -- higher-level protocol clients built on the transport layer -- shell and debugging tools -- more board demos and deployment examples -- additional scheduler tests and edge-case coverage +## Documentation + +The [smallOS guide](guide/README.md) is the main documentation entry point. + +- [Installation and validation](guide/installation.md) +- [Quick start](guide/quick-start.md) +- [Runtime concepts](guide/runtime-concepts.md) +- [Task lifecycle and coordination](guide/task-lifecycle.md) +- [Configuration](guide/configuration.md) +- [Core API reference](guide/api-reference.md) +- [Error handling and scheduler wakeups](guide/error-handling.md) +- [Kernels and MicroPython](guide/kernels-and-micropython.md) +- [Execution adapters](guide/execution-adapters.md) +- [Networking clients](guide/networking-clients.md) +- [Shell and TCP servers](guide/shell-and-server.md) +- [Demos](guide/demos.md) +- [Troubleshooting and limitations](guide/troubleshooting.md) +- [Contributing](guide/contributing.md) + +## Project layout + +- [`SmallPackage/`](SmallPackage): runtime package +- [`SmallPackage/clients/`](SmallPackage/clients): cooperative protocol clients +- [`SmallPackage/adapters/`](SmallPackage/adapters): blocking and asyncio bridges +- [`demos/`](demos): desktop and board examples +- [`tests/`](tests): unit, integration, and typing contracts ## License -This project is licensed under the MIT License. See -[LICENSE](LICENSE). +smallOS is licensed under the [MIT License](LICENSE). diff --git a/demo.py b/demo.py index 1508259..7651ec9 100644 --- a/demo.py +++ b/demo.py @@ -11,6 +11,9 @@ DEMO_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "demos") if DEMO_DIR not in sys.path: + # demos/common.py is intentionally imported as a sibling by runnable demo + # files. Adding this directory preserves the original root command without + # duplicating the showcase implementation. sys.path.insert(0, DEMO_DIR) diff --git a/demos/adapters_asyncio_demo.py b/demos/adapters_asyncio_demo.py index 7d1ad97..bf73caf 100644 --- a/demos/adapters_asyncio_demo.py +++ b/demos/adapters_asyncio_demo.py @@ -27,6 +27,8 @@ def _check_loop(self) -> int: return loop_id async def open(self) -> int: + # These resources are created on the adapter loop, never on the smallOS + # scheduler thread or a temporary asyncio.run() loop. self.owner_loop_id = self._check_loop() self.queue = asyncio.Queue() self.worker_task = asyncio.create_task(self._run()) @@ -71,6 +73,8 @@ async def asyncio_example( service: AsyncioWorker, ) -> tuple[str, str]: """Create and reuse a loop-affine service across adapter calls.""" + # Separate calls reuse the adapter's one persistent loop, which is required + # by queues, futures, clients, and background tasks with loop affinity. opened_loop = await adapter.call(service.open) try: first, first_loop = await adapter.call(service.process, "smallos") @@ -89,6 +93,8 @@ def main() -> None: runtime = build_runtime(Unix()) service = AsyncioWorker() + # The context manager stays open until all SmallOS work using the adapter is + # finished, then tears down the foreign loop deterministically. with AsyncioAdapter(max_pending=8) as foreign_async: target = SmallTask( 2, diff --git a/demos/adapters_demo.py b/demos/adapters_demo.py index 2f07600..dd82055 100644 --- a/demos/adapters_demo.py +++ b/demos/adapters_demo.py @@ -28,6 +28,8 @@ async def blocking_adapter_demo( task: SmallTask[str], adapter: ThreadAdapter, ) -> str: + # adapter.call emits a smallOS-owned instruction. The worker thread never + # mutates scheduler queues or resumes this task directly. result = await adapter.call(blocking_library_call, "SmallOS") task_runtime(task).print(result + "\n") return result @@ -37,6 +39,8 @@ async def asyncio_adapter_demo( task: SmallTask[str], adapter: AsyncioAdapter, ) -> str: + # The callable runs on the adapter's persistent asyncio loop, then its result + # returns through a readiness object watched by the smallOS kernel. result = await adapter.call(asyncio_library_call, "SmallOS") task_runtime(task).print(result + "\n") return result @@ -51,6 +55,8 @@ async def cooperative_peer(task: SmallTask[str]) -> str: def main() -> None: runtime = build_runtime(Unix()) + # Adapter lifetime surrounds runtime.start(): shutdown is explicit, and a + # live adapter binds to the first SmallOS runtime that submits work to it. with ThreadAdapter(max_workers=2, max_pending=8) as blocking: with AsyncioAdapter(max_pending=8) as foreign_async: runtime.fork( diff --git a/demos/adapters_sqlite_demo.py b/demos/adapters_sqlite_demo.py index 9fea534..c00daef 100644 --- a/demos/adapters_sqlite_demo.py +++ b/demos/adapters_sqlite_demo.py @@ -20,6 +20,8 @@ def __init__(self) -> None: self.owner_thread_id: int | None = None def open(self) -> None: + # SQLite's default connection enforces same-thread use. Record the lane + # owner so this demo turns an ownership mistake into a clear exception. self.owner_thread_id = threading.get_ident() self.connection = sqlite3.connect(":memory:") self.connection.execute( @@ -60,6 +62,9 @@ async def sqlite_example( store: SQLiteStore, ) -> list[tuple[int, str]]: """Create, use, and close SQLite entirely on the adapter worker.""" + # Creation, every operation, and close all traverse the same one-worker lane. + # Creating the connection on the smallOS scheduler thread would violate the + # ownership contract as soon as the adapter tried to use it. await adapter.call(store.open) try: await adapter.call( @@ -77,7 +82,8 @@ def main() -> None: runtime = build_runtime(Unix()) store = SQLiteStore() - # One worker creates a serialized execution lane for this connection. + # One worker creates a serialized execution lane for this connection; + # max_pending bounds queued work without blocking the scheduler. with ThreadAdapter(max_workers=1, max_pending=8) as blocking: target = SmallTask( 2, diff --git a/demos/common.py b/demos/common.py index 6a70455..fd2cbc8 100644 --- a/demos/common.py +++ b/demos/common.py @@ -25,6 +25,8 @@ CONFIG_PATH = os.path.join(REPO_ROOT, "smallos.config.json") +# Signals are integer slots. A named constant gives application-level meaning +# to a slot without making that meaning part of the smallOS scheduler. DEMO_SIGNAL = 3 @@ -38,6 +40,9 @@ def load_demo_config(**overrides): def build_runtime(kernel: Kernel, **config_overrides: Any) -> SmallOS: """Create a ``SmallOS`` instance wired to the chosen kernel.""" + # The kernel owns platform behavior; SmallOS owns task scheduling. Keeping + # this attachment explicit is what lets the same task code use Unix or a + # MicroPython board profile. runtime = SmallOS(config=load_demo_config(**config_overrides)).setKernel(kernel) return install_demo_error_handler(runtime) @@ -93,6 +98,8 @@ def install_demo_error_handler(runtime, include_cancelled=False): """Attach the shared demo error logger to ``runtime``.""" def _handler(event): + # Error handlers are synchronous observers. They report a task after + # finalization and must not try to drive coroutine work themselves. runtime.kernel.write(_format_failure_event(event)) runtime.setErrorHandler(_handler, include_cancelled=include_cancelled) @@ -110,6 +117,8 @@ async def worker(task): async def join_demo(task): """Show child spawning plus ordered ``join_all`` collection.""" task.OS.print("join demo starting\n") + # Smaller priority numbers are scheduled first. The join result below is + # nevertheless returned in this caller-supplied order. fast = task.spawn(worker, priority=1, name="fast") medium = task.spawn(worker, priority=3, name="medium") slow = task.spawn(worker, priority=5, name="slow") @@ -130,6 +139,7 @@ async def signal_demo(task): """Show a task blocked on a signal and then joined with its sender.""" task.OS.print("signal demo waiting\n") sender = task.spawn(signal_sender, priority=max(1, task.priority - 1), name="signal_sender") + # wait_signal suspends this task; it does not block the scheduler thread. signal = await task.wait_signal(DEMO_SIGNAL) sender_result = await task.join(sender) task.OS.print("signal demo resumed on {} with {}\n".format(signal, sender_result)) @@ -145,6 +155,8 @@ async def startup_banner(task, board_name): def default_tasks(board_name): """Return a small starter task set used by most demos.""" + # Each SmallTask wraps an async routine. fork() later assigns PIDs and puts + # these ready tasks into their per-priority FIFO queues. return [ SmallTask(2, startup_banner, name="startup_banner", args=(board_name,)), SmallTask(4, signal_demo, name="signal_demo"), diff --git a/demos/esp32_demo.py b/demos/esp32_demo.py index 0890421..5b08e35 100644 --- a/demos/esp32_demo.py +++ b/demos/esp32_demo.py @@ -22,7 +22,11 @@ def maybe_connect_wifi(kernel): def main(): + # The profile centralizes board-specific Wi-Fi, timing, socket, and polling + # behavior so the tasks below do not import MicroPython modules directly. kernel = ESP32(hostname=WIFI_HOSTNAME) + # Leave credentials as None when network access is not needed. Do not commit + # real secrets to a demo; inject them through your deployment workflow. maybe_connect_wifi(kernel) runtime = build_runtime(kernel) diff --git a/demos/http_demo.py b/demos/http_demo.py index a6fd4fb..28f4ef3 100644 --- a/demos/http_demo.py +++ b/demos/http_demo.py @@ -9,7 +9,10 @@ async def http_demo(task): + # Passing the attached task lets the client inherit transport limits from + # task.OS.config and suspend on this runtime's kernel readiness operations. client = SmallHTTPClient(task, base_url=HTTP_BASE_URL) + # While connect/send/receive waits for the socket, other smallOS tasks may run. response = await client.get("/", headers={"Accept": "text/html"}) preview = response.text().replace("\n", " ")[:120] task.OS.print("http status: {} {}\n".format(response.status_code, response.reason)) @@ -19,6 +22,7 @@ async def http_demo(task): def main(): runtime = build_runtime(Unix()) + # Priority 2 is a scheduler category; lower numeric categories run first. runtime.fork([SmallTask(2, http_demo, name="http_demo")]) runtime.startOS() diff --git a/demos/micropython_autodetect_demo.py b/demos/micropython_autodetect_demo.py index c53f21c..c441390 100644 --- a/demos/micropython_autodetect_demo.py +++ b/demos/micropython_autodetect_demo.py @@ -16,6 +16,9 @@ def maybe_connect_wifi(kernel): def main(): + # Detection reads the firmware machine string and returns a matching built-in + # profile. Explicit ESP32/PicoW construction is preferable when an app needs + # profile-specific settings. kernel = build_micropython_kernel() maybe_connect_wifi(kernel) diff --git a/demos/mqtt_demo.py b/demos/mqtt_demo.py index 7889cfc..0bf8e19 100644 --- a/demos/mqtt_demo.py +++ b/demos/mqtt_demo.py @@ -13,6 +13,8 @@ async def mqtt_demo(task): + # Point these settings at a broker you control. QoS 1 is used so the demo + # visibly exercises acknowledgement handling rather than fire-and-forget. client = SmallMQTTClient( task, host=MQTT_HOST, @@ -21,6 +23,7 @@ async def mqtt_demo(task): client_id="smallos-demo-client", ) await client.connect() + # Every network await yields to smallOS while the broker socket is not ready. suback = await client.subscribe(MQTT_TOPIC, qos=MQTT_QOS) publish_info = await client.publish(MQTT_TOPIC, "hello from smallOS", qos=MQTT_QOS) task.OS.print("mqtt subscribed to {} with granted QoS {}\n".format(MQTT_TOPIC, suback["granted_qos"])) @@ -30,6 +33,7 @@ async def mqtt_demo(task): task.OS.print( "mqtt received {} -> {} at QoS {}\n".format(message["topic"], message["payload"], message["qos"]) ) + # A clean MQTT disconnect is part of protocol cleanup, not just socket close. await client.disconnect() return message diff --git a/demos/pico_w_demo.py b/demos/pico_w_demo.py index dc31cd8..38a016d 100644 --- a/demos/pico_w_demo.py +++ b/demos/pico_w_demo.py @@ -24,11 +24,14 @@ def maybe_connect_wifi(kernel): def main(): + # Country and power-management settings are profile options because their + # implementation is firmware/board specific, not scheduler policy. kernel = PicoW( country=WIFI_COUNTRY, hostname=WIFI_HOSTNAME, power_management=WIFI_POWER_MANAGEMENT, ) + # Keep real credentials outside source control in an application deployment. maybe_connect_wifi(kernel) runtime = build_runtime(kernel) diff --git a/demos/redis_demo.py b/demos/redis_demo.py index 6051d47..893ca73 100644 --- a/demos/redis_demo.py +++ b/demos/redis_demo.py @@ -11,6 +11,8 @@ async def redis_demo(task): + # The native client speaks RESP over a cooperative SmallStream. It does not + # start an asyncio loop or a background thread. client = SmallRedisClient( task, host=REDIS_HOST, @@ -23,6 +25,7 @@ async def redis_demo(task): value = await client.get("smallos:demo") task.OS.print("redis ping: {}\n".format(pong)) task.OS.print("redis value: {}\n".format(value)) + # This task created the connection, so it also owns deterministic cleanup. client.close() return value diff --git a/demos/runtime_demo.py b/demos/runtime_demo.py index e30aba5..a0e087d 100644 --- a/demos/runtime_demo.py +++ b/demos/runtime_demo.py @@ -19,6 +19,8 @@ async def priority_worker(task): """Simple child workload used to show priority-aware interleaving.""" for step in range(3): task.OS.print("[{}] step {}\n".format(task.name, step)) + # sleep() records a monotonic wake deadline and lets another ready task + # run; it never calls blocking time.sleep(). await task.sleep(0.05) return task.name @@ -26,9 +28,13 @@ async def priority_worker(task): async def join_demo(task): """Spawn three workers and collect their results in a fixed order.""" task.OS.print("join demo starting\n") + # spawn() establishes the parent/child relationship and returns the child + # object, which can be passed directly to join() or join_all(). fast = task.spawn(priority_worker, priority=1, name="fast") medium = task.spawn(priority_worker, priority=3, name="medium") slow = task.spawn(priority_worker, priority=5, name="slow") + # Completion timing may differ, but join_all preserves this requested order. + # A child exception would instead be raised into this parent task. results = await task.join_all([fast, medium, slow]) task.OS.print("join demo results: {}\n".format(results)) return results @@ -48,6 +54,8 @@ async def http_request_task(task, base_url=HTTP_BASE_URL, path=HTTP_PATH): async def http_request_demo(task): """Show a network request running while the parent keeps doing work.""" task.OS.print("http request demo starting\n") + # The child handles socket readiness while this parent continues independent + # cooperative work. No operating-system thread is created for the request. request = task.spawn( http_request_task, priority=max(1, task.priority - 1), @@ -59,6 +67,7 @@ async def http_request_demo(task): task.OS.print("http request parent doing other work {}\n".format(step)) await task.sleep(0.05) + # If the HTTP child failed, join() would raise that exception here. response = await task.join(request) task.OS.print("http request status: {} {}\n".format(response["status"], response["reason"])) task.OS.print("http request preview: {}\n".format(response["preview"])) @@ -69,6 +78,8 @@ async def signal_sender(task): """Sleep for a while and then wake the parent by sending a signal.""" await task.sleep(0.1) task.OS.print("sender raising signal 3\n") + # Signals are latched integer slots. Sending before the parent reaches its + # wait is safe because wait_signal() consumes an already-latched signal. task.sendSignal(task.parent.pid, 3) return "signal sent" @@ -87,12 +98,18 @@ async def cooperative_demo(task): """Show a task voluntarily yielding without waiting on time or signals.""" for index in range(5): task.OS.print("cooperative tick {}\n".format(index)) + # yield_now() remains immediately runnable but gives the priority queues + # another scheduling opportunity. await task.yield_now() return "done" def main(): + # build_runtime installs the Unix kernel, repository config, and a default + # task-failure observer shared by all demos. runtime = build_runtime(Unix()) + # These are top-level peers. Lower numeric priorities are considered first; + # awaits still allow lower-priority work to make progress while peers wait. runtime.fork( [ SmallTask(2, http_request_demo, name="http_request_demo"), diff --git a/demos/shell_demo.py b/demos/shell_demo.py index 88f99aa..cd09809 100644 --- a/demos/shell_demo.py +++ b/demos/shell_demo.py @@ -30,6 +30,8 @@ async def shell_session(task): still uses the same command parser and runtime APIs as an interactive shell would. """ + # The shell is already attached to this runtime. Commands below call the + # same APIs an interactive stdin-backed shell uses. shell = task.OS.shells[0] worker_pid = _pid_for_name(task.OS, "background_worker") script = [ @@ -45,6 +47,8 @@ async def shell_session(task): ] for command in script: + # Sleeping between commands proves background tasks continue to advance + # instead of a blocking input loop owning the process. await task.sleep(0.05) shell.run(command, show_prompt=False, echo_command=True, force_output=True) if not shell.is_running: @@ -53,9 +57,13 @@ async def shell_session(task): def main(): + # allow_python defaults to True for a local debugging shell. Disable it for + # any input source that is not fully trusted. shell = BaseShell() runtime = build_runtime(Unix()) runtime.shells.append(shell.setOS(runtime)) + # The scripted session is a normal task, so its output and priority interact + # with application work through the same scheduler. runtime.fork( [ SmallTask(2, shell_session, name="shell_session"), diff --git a/demos/unix_demo.py b/demos/unix_demo.py index 233e46a..2ee52df 100644 --- a/demos/unix_demo.py +++ b/demos/unix_demo.py @@ -6,8 +6,12 @@ def main(): + # Unix supplies desktop timing, terminal output, sockets, and selector-based + # readiness while the runtime remains responsible for task ordering. runtime = build_runtime(Unix()) + # Registration assigns PIDs but does not execute user routines yet. runtime.fork(default_tasks("Unix")) + # startOS() drives the scheduler until no non-watcher work remains. runtime.startOS() diff --git a/demos/web_app_demo.py b/demos/web_app_demo.py index bb1c37a..8a07ebc 100644 --- a/demos/web_app_demo.py +++ b/demos/web_app_demo.py @@ -190,6 +190,8 @@ async def _send_all(task, sock, data): try: sent = kernel.socket_send(sock, remaining) except Exception as exc: + # Retry direction is operation-aware. TLS and some socket errors may + # require waiting for the opposite direction from the operation name. retry_mode = kernel.socket_retry_mode(exc, "send") if retry_mode == "read": await task.wait_readable(sock) @@ -271,6 +273,8 @@ async def web_client_handler(task, client_sock, client_addr, state): state["active_connections"] += 1 task.OS.print("Accepted connection from {} (active connections: {})\n".format(client_addr, state["active_connections"])) + # The handler owns this accepted socket from dispatch through the finally + # block. Keeping one close owner prevents descriptor leaks and double-close. try: try: request_head = await _read_request_head(task, client_sock) @@ -355,6 +359,8 @@ def _open_listener(kernel, host, port, backlog): if not kernel.supports_tcp_server(): raise NotImplementedError("This kernel does not support passive TCP servers.") + # The resolved address record is intentionally opaque. Passing it unchanged + # keeps tuple layout and address-family details inside the active kernel. address_info = kernel.resolve_passive_address(host, port) listener = kernel.socket_open(address_info) try: @@ -374,6 +380,8 @@ def _dispatch_client(task, client_sock, client_addr, state): kernel = task.OS.kernel try: kernel.socket_setblocking(client_sock, False) + # After spawn succeeds the handler owns the stream. Before that point, + # this function must roll it back if task registration fails. task.spawn( web_client_handler, priority=max(1, task.priority - 1), @@ -401,6 +409,9 @@ async def web_server_task(task, state): try: client_sock, client_addr = kernel.socket_accept(listener) except Exception as exc: + # A non-blocking listener normally reaches this path until a + # connection arrives. The readiness await keeps the one-thread + # scheduler free to run metrics, state, shell, and client tasks. retry_mode = kernel.socket_retry_mode(exc, "accept") if retry_mode == "read": await task.wait_readable(listener) @@ -419,6 +430,8 @@ async def web_server_task(task, state): def main(): """Start the demo runtime and keep serving until interrupted.""" runtime = build_runtime(Unix()) + # The interactive shell can cancel/inspect server tasks. Python evaluation + # is disabled because it is unnecessary for this application demo. shell = BaseShell(prompt="webapp> ", allow_python=False) runtime.shells.append(shell.setOS(runtime)) @@ -431,6 +444,9 @@ def main(): "demo_value": 0, } + # The listener receives the highest priority here. Maintenance loops are + # watchers, so the default runtime policy will not keep them alive after the + # server and shell work has ended. web_server = SmallTask(2, web_server_task, name="web_server", args=(state,)) shell_stdin = shell.make_task( priority=3, diff --git a/guide/README.md b/guide/README.md new file mode 100644 index 0000000..ae3c106 --- /dev/null +++ b/guide/README.md @@ -0,0 +1,34 @@ +# smallOS Guide + +This guide covers installation, the runtime model, platform kernels, library +integration, and the built-in networking helpers. If this is your first time +using smallOS, follow the first three pages in order. + +## Start here + +1. [Installation and validation](installation.md) +2. [Quick start](quick-start.md) +3. [Runtime concepts](runtime-concepts.md) + +## Build an application + +- [Task lifecycle and coordination](task-lifecycle.md) +- [Configuration](configuration.md) +- [Core API reference](api-reference.md) +- [Error handling and scheduler wakeups](error-handling.md) +- [Kernels and MicroPython](kernels-and-micropython.md) +- [Execution adapters](execution-adapters.md) +- [Networking clients](networking-clients.md) +- [Shell and TCP servers](shell-and-server.md) +- [Demos](demos.md) +- [Troubleshooting and limitations](troubleshooting.md) + +## Work on smallOS + +- [Contributing](contributing.md) +- [Project source](../SmallPackage) +- [Test suite](../tests) + +smallOS is currently experimental. Its core runtime is usable, but APIs may +continue to evolve while desktop behavior and MicroPython portability are +expanded. diff --git a/guide/api-reference.md b/guide/api-reference.md new file mode 100644 index 0000000..66afa60 --- /dev/null +++ b/guide/api-reference.md @@ -0,0 +1,122 @@ +# Core API Reference + +[Previous: Configuration](configuration.md) · [Guide home](README.md) · +[Next: Error handling](error-handling.md) + +This page summarizes the public application-facing surface. Internal queue, +resume, completion, and registration methods are scheduler implementation +details even when Python does not enforce privacy. + +## `SmallOS` + +### Construction + +```python +SmallOS(size=None, config=None, **overrides) +``` + +- `config` accepts `SmallOSConfig`, a compatible dictionary, or `None`. +- `size` overrides `task_capacity` for compatibility. +- `priority_levels`, `io_buffer_length`, and `eternal_watchers` may be passed as + constructor overrides. + +### Application methods + +| Method | Result | Purpose | +| --- | --- | --- | +| `setKernel(kernel)` | runtime | Attach timing, output, and I/O primitives. | +| `fork(task)` | PID | Register one top-level task. | +| `fork([tasks])` | list of PIDs | Register tasks in input order. | +| `start()` | `None` | Run until no live work remains under watcher policy. | +| `startOS()` | `None` | Compatibility alias for `start()`. | +| `setErrorHandler(handler, include_cancelled=False)` | runtime | Observe finalized task failures. | +| `setEternalWatchers(enabled)` | runtime | Change watcher-only exit behavior. | +| `cancel_task(task_or_pid, recursive=False)` | `0` or `-1` | Cancel a registered task. | +| `print(...)` | `None` | Write application output through `SmallIO`. | + +## `SmallTask` + +### Construction + +```python +SmallTask(priority, routine, name="", args=(), isReady=1, isWatcher=False) +``` + +The routine receives the task object first. `args` may be positional, keyword, +or a single extra value as described in [Task lifecycle](task-lifecycle.md). + +### Properties and methods + +| Member | Purpose | +| --- | --- | +| `done` | Whether the task reached a terminal state. | +| `result` | Stored successful result, otherwise `None`. | +| `exception` | Stored terminal exception, otherwise `None`. | +| `getID()` | Return the assigned PID (`-1` before registration). | +| `spawn(routine_or_task, priority=None, **kwargs)` | Register and return a child task. | +| `join(child)` | Return an awaitable for one child result. | +| `join_all(children)` | Return an awaitable for ordered results. | +| `sleep(seconds)` | Cooperatively wait for time. | +| `yield_now()` | Return to the ready queue. | +| `wait_signal(sig)` | Wait for signal slot `0`–`31`. | +| `sendSignal(pid, sig)` | Deliver a signal through the owning runtime. | +| `wait_readable(obj)` | Wait for kernel read readiness. | +| `wait_writable(obj)` | Wait for kernel write readiness. | +| `getSignals()` | List currently latched signal numbers. | + +Prefer `runtime.cancel_task(...)` over direct lifecycle mutation methods so the +scheduler can remove wait registrations and notify joiners consistently. + +## `SmallOSConfig` + +| Method | Purpose | +| --- | --- | +| `SmallOSConfig.default()` | Return a fresh default configuration. | +| `from_dict(data)` | Load canonical fields and supported aliases. | +| `from_json_file(path)` | Load JSON using `json` or `ujson`. | +| `copy(**updates)` | Create an updated independent configuration. | +| `to_dict()` | Produce plain serializable data. | +| `client_defaults_for(section)` | Merge stream defaults with one client section. | + +See [Configuration](configuration.md) for fields and defaults. + +## Kernels + +Application code normally selects a built-in kernel and uses task awaitables +rather than calling low-level socket methods directly: + +```python +runtime.setKernel(Unix()) +``` + +Portable components should check optional capabilities before use: + +- `supports_tcp_server()` for passive listeners +- `supports_reuse_address()` for listener address reuse +- `supports_wakeup_channel()` for cross-thread scheduler notifications +- `supports_external_wait_objects()` before execution adapters depend on + readiness objects + +The [kernel guide](kernels-and-micropython.md) explains profiles and the +[shell/server guide](shell-and-server.md) shows the passive TCP sequence. + +## Execution adapters and clients + +Adapters and protocol clients have larger, subsystem-specific APIs: + +- [Execution adapters](execution-adapters.md) +- [Networking clients](networking-clients.md) +- [Detailed client reference](../SmallPackage/clients/README.md) + +## Exceptions + +Common application-visible failures include: + +- `TaskCancelledError`: scheduler-owned cancellation +- `UnsupportedAwaitableError`: an awaited object is not a smallOS instruction +- `MaxProcessError`: capacity or task registration failed +- `AdapterCapacityError`: adapter `max_pending` was reached +- protocol-specific client errors exported from `SmallPackage` + +Install a runtime [error handler](error-handling.md) so failures in unjoined +top-level tasks remain visible. diff --git a/guide/configuration.md b/guide/configuration.md new file mode 100644 index 0000000..05e52ab --- /dev/null +++ b/guide/configuration.md @@ -0,0 +1,67 @@ +# Configuration + +[Previous: Task lifecycle](task-lifecycle.md) · [Guide home](README.md) · +[Next: API reference](api-reference.md) + +The runtime uses `SmallOSConfig`, which can be constructed directly or loaded +from [`smallos.config.json`](../smallos.config.json). + +```python +from SmallPackage import SmallOS, SmallOSConfig + +config = SmallOSConfig.from_json_file("smallos.config.json") +runtime = SmallOS(config=config) +``` + +## Fields + +- `task_capacity`: maximum tracked tasks and PID slots +- `priority_levels`: number of ready-queue categories +- `io_buffer_length`: buffered application output retained when terminal view + is hidden +- `eternal_watchers`: whether watcher-only work keeps the runtime alive +- `client_defaults`: shared stream and protocol-client limits + +Example: + +```json +{ + "task_capacity": 1024, + "priority_levels": 10, + "io_buffer_length": 1024, + "eternal_watchers": false, + "client_defaults": { + "stream": { + "max_buffer_size": 16777216 + }, + "http": { + "max_response_size": 16777216 + }, + "redis": { + "max_response_size": 16777216, + "max_nesting_depth": 32 + }, + "mqtt": { + "keepalive": 60, + "max_packet_size": 262144, + "max_queued_messages": 1024 + }, + "sse": { + "max_event_size": 1048576, + "max_line_size": 65536 + }, + "websocket": { + "max_frame_size": 1048576, + "max_message_size": 4194304, + "max_line_size": 16384 + } + } +} +``` + +The loader also accepts `oslist_length` as an alias for `task_capacity` and +`num_categories` as an alias for `priority_levels` so older experiments can +map onto the current runtime. + +Client constructors accept explicit overrides. When a client is created from +an attached task, omitted values inherit from `task.OS.config`. diff --git a/guide/contributing.md b/guide/contributing.md new file mode 100644 index 0000000..13eab9a --- /dev/null +++ b/guide/contributing.md @@ -0,0 +1,44 @@ +# Contributing + +[Previous: Troubleshooting](troubleshooting.md) · [Guide home](README.md) + +Contributions, issues, experiments, and board-port notes are welcome. Useful +areas include: + +- validating MicroPython ports on real boards +- protocol clients built on the portable transport layer +- shell and debugging tools +- board demos and deployment examples +- scheduler tests and edge-case coverage +- documentation corrections and examples + +## Development checks + +Install the development dependencies as described in +[Installation](installation.md), then run checks proportional to the change: + +```bash +python -m unittest discover -s tests -v +coverage run -m unittest discover -s tests -v +coverage report +pyright +python -m build +``` + +The CI pipeline tests CPython 3.10 through 3.13 and verifies distributions. +When changing core abstractions, account for both desktop CPython and +MicroPython-oriented imports and demos. + +## Repository map + +- [`SmallPackage/SmallOS.py`](../SmallPackage/SmallOS.py): scheduler +- [`SmallPackage/SmallTask.py`](../SmallPackage/SmallTask.py): coroutine stepping and joins +- [`SmallPackage/Kernel.py`](../SmallPackage/Kernel.py): platform kernels +- [`SmallPackage/SmallIO.py`](../SmallPackage/SmallIO.py): output routing +- [`SmallPackage/SmallConfig.py`](../SmallPackage/SmallConfig.py): runtime configuration +- [`SmallPackage/clients/`](../SmallPackage/clients): protocol clients +- [`SmallPackage/adapters/`](../SmallPackage/adapters): foreign execution bridges +- [`demos/`](../demos): examples +- [`tests/`](../tests): test suite + +smallOS is licensed under the [MIT License](../LICENSE). diff --git a/guide/demos.md b/guide/demos.md new file mode 100644 index 0000000..d01e00d --- /dev/null +++ b/guide/demos.md @@ -0,0 +1,45 @@ +# Demos + +[Previous: Shell and TCP servers](shell-and-server.md) · [Guide home](README.md) · +[Next: Troubleshooting](troubleshooting.md) + +The [`demos/`](../demos) directory contains focused runnable examples. + +## Runtime and shells + +- [`unix_demo.py`](../demos/unix_demo.py): desktop scheduler +- [`runtime_demo.py`](../demos/runtime_demo.py): broader runtime showcase +- [`shell_demo.py`](../demos/shell_demo.py): scripted shell beside cooperative tasks + +The root [`demo.py`](../demo.py) remains a compatibility wrapper around the +runtime demo. + +## Boards + +- [`esp32_demo.py`](../demos/esp32_demo.py): ESP32 startup and optional Wi-Fi +- [`pico_w_demo.py`](../demos/pico_w_demo.py): Pico W startup and Wi-Fi +- [`micropython_autodetect_demo.py`](../demos/micropython_autodetect_demo.py): + automatic kernel selection + +## Networking + +- [`http_demo.py`](../demos/http_demo.py): native HTTP client +- [`redis_demo.py`](../demos/redis_demo.py): native Redis client +- [`mqtt_demo.py`](../demos/mqtt_demo.py): native MQTT client +- [`web_app_demo.py`](../demos/web_app_demo.py): single-thread HTTP routes, live + browser UI, and shell-driven shutdown + +## Execution adapters + +- [`adapters_demo.py`](../demos/adapters_demo.py): both adapters beside a smallOS task +- [`adapters_sqlite_demo.py`](../demos/adapters_sqlite_demo.py): thread-affine SQLite +- [`adapters_asyncio_demo.py`](../demos/adapters_asyncio_demo.py): persistent asyncio resources + +Shared entry points install a default error handler through +[`demos/common.py`](../demos/common.py). Network failures, invalid I/O wait +objects, and other uncaught task exceptions therefore appear as readable task +diagnostics. + +The demos contain inline comments explaining where control returns to the +scheduler, why resource ownership matters, and how kernel capability checks +keep the same application shape portable. diff --git a/guide/error-handling.md b/guide/error-handling.md new file mode 100644 index 0000000..cfad5c6 --- /dev/null +++ b/guide/error-handling.md @@ -0,0 +1,79 @@ +# Error Handling and Scheduler Wakeups + +[Previous: API reference](api-reference.md) · [Guide home](README.md) · +[Next: Kernels and MicroPython](kernels-and-micropython.md) + +## Observe task failures + +Install a runtime-level synchronous error observer with +`setErrorHandler(handler, include_cancelled=False)`: + +```python +from SmallPackage import SmallOS, Unix + + +def log_runtime_error(event): + print( + "[smallOS] task failure in {} (PID {}): {}".format( + event["task_name"] or "unnamed task", + event["task_id"], + event["exception_repr"], + ) + ) + if event["traceback_text"]: + print(event["traceback_text"], end="") + + +runtime = SmallOS().setKernel(Unix()) +runtime.setErrorHandler(log_runtime_error) +``` + +The observer runs after the failed task is finalized. Events include task, +parent, exception, cancellation, blocked/waiting, join, adapter, and traceback +details. The currently exposed keys are: + +- `task_id`, `task_name`, and `parent_id` +- `exception`, `exception_type`, and `exception_repr` +- `is_cancelled` and `traceback_text` +- `blocked_reason`, `waiting_signal`, and `io_wait_mode` +- `join_target_id` and `join_pending_ids` +- `adapter_name` and `adapter_job_id` + +Cancellation events are excluded by default. Pass `include_cancelled=True` to +observe `TaskCancelledError` as well. + +## Invalid I/O objects + +A closed or invalid file descriptor passed to `wait_readable(...)` or +`wait_writable(...)` fails the waiting task instead of crashing the scheduler +through the platform poll/select layer. The runtime finalizes that task and +delivers its exception—commonly `ValueError`—to the error observer. + +## Wake a blocked scheduler from another thread + +Some kernels expose an opaque wakeup channel for external threads that need to +request scheduler-owned work, such as server shutdown: + +```python +kernel = Unix() +if not kernel.supports_wakeup_channel(): + raise RuntimeError("cross-thread scheduler wakeup is unavailable") + +wakeup = kernel.create_wakeup_channel() + + +async def watch_shutdown(task): + await task.wait_readable(wakeup.wait_object) + wakeup.drain() + # Apply the application-owned request on the scheduler thread. +``` + +Call `wakeup.notify()` from the external thread. Notifications are nonblocking +and coalesce until the scheduler calls `drain()`. Call `close()` only after the +scheduler wait is detached; repeated close, notify, and drain calls are safe +during teardown. + +`Unix` supports this contract when `socketpair()` is available. Generic +`MicroPythonKernel` deliberately reports it unsupported because a polling +backend alone does not establish safe cross-thread behavior on a constrained +port. Scheduler-native shutdown does not need a cross-thread wakeup channel. diff --git a/guide/execution-adapters.md b/guide/execution-adapters.md new file mode 100644 index 0000000..46d46cd --- /dev/null +++ b/guide/execution-adapters.md @@ -0,0 +1,99 @@ +# Execution Adapters + +[Previous: Kernels and MicroPython](kernels-and-micropython.md) · +[Guide home](README.md) · [Next: Networking clients](networking-clients.md) + +Execution adapters let a smallOS task yield while user-supplied code runs under +a different execution model. They use the standard library and do not install, +configure, or wrap database drivers, ORMs, SDKs, or other packages. + +## Synchronous blocking code + +Use `ThreadAdapter` for a blocking callable: + +```python +from SmallPackage import SmallTask +from SmallPackage.adapters.threads import ThreadAdapter + + +def load_record(user_library, settings, record_id): + connection = user_library.connect(**settings) + try: + return connection.load(record_id) + finally: + connection.close() + + +with ThreadAdapter(max_workers=4, max_pending=64) as blocking: + async def load(task): + return await blocking.call( + load_record, + user_selected_library, + connection_settings, + 42, + ) + + runtime.fork(SmallTask(2, load, name="load")) + runtime.start() +``` + +Use `max_workers=1` when a connection or session must remain on one serialized, +thread-affine lane. Create, use, and close that resource through calls on the +same adapter. + +## Asyncio-owned code + +`AsyncioAdapter` owns one persistent event loop on a dedicated thread, so +loop-affine resources can be reused when all operations use the same adapter: + +```python +from SmallPackage import SmallTask +from SmallPackage.adapters.asyncio_loop import AsyncioAdapter + + +async def fetch_record(user_library, settings, record_id): + async with user_library.Client(**settings) as client: + return await client.fetch(record_id) + + +with AsyncioAdapter(max_pending=64) as foreign_async: + async def load(task): + return await foreign_async.call( + fetch_record, + user_selected_async_library, + connection_settings, + 42, + ) + + runtime.fork(SmallTask(2, load, name="load")) + runtime.start() +``` + +Pass an async callable to `call()`, not an asyncio `Task` or `Future` already +owned by another loop. + +## Lifecycle and cancellation + +- An adapter binds to the first smallOS runtime that uses it. +- Shutdown is explicit; wrap `runtime.start()` in the context manager. +- `max_pending` rejects excess work with `AdapterCapacityError` rather than + blocking the scheduler. +- Cancelling a smallOS task can cancel queued thread work but cannot forcibly + stop a running Python thread. +- Asyncio cancellation is requested on the adapter loop, but foreign code may + delay or suppress it. +- `AsyncioAdapter.shutdown_error` records an unexpected loop stop or teardown + failure; normal shutdown leaves it as `None`. + +## Standard-library examples + +These demos require nothing beyond smallOS: + +```bash +python demos/adapters_demo.py +python demos/adapters_sqlite_demo.py +python demos/adapters_asyncio_demo.py +``` + +The SQLite demo owns its connection on one thread worker. The asyncio demo +reuses a queue, futures, and a background task on the persistent adapter loop. diff --git a/guide/installation.md b/guide/installation.md new file mode 100644 index 0000000..3a64c34 --- /dev/null +++ b/guide/installation.md @@ -0,0 +1,137 @@ +# Installation and Validation + +[Guide home](README.md) · [Next: Quick start](quick-start.md) + +## Requirements + +smallOS supports CPython 3.10 and newer. Python 3.6 through 3.9 are no longer +supported. MicroPython compatibility is maintained separately because its +language and standard-library support do not map directly to a CPython release. + +The runtime has no third-party dependencies. + +## Choose an installation path + +> **Do not run `pip install SmallPackage`.** The normalized PyPI name +> `smallpackage` belongs to an unrelated project. smallOS currently publishes +> verified wheel and source archives through GitHub Releases, not PyPI. + +### Install from a GitHub Release + +Download the wheel for the desired version from the project's +[GitHub Releases](https://github.com/MikiEEE/SmallOS/releases), then install the +local file: + +```bash +python -m pip install /path/to/smallpackage-VERSION-py3-none-any.whl +``` + +Confirm that the intended package is importable: + +```bash +python -c "from SmallPackage import SmallOS, SmallTask, Unix; print('smallOS ready')" +``` + +The wheel is platform-independent Python code, but execution adapters require +desktop CPython facilities such as threads, sockets, and asyncio. + +### Install from a source checkout + +For a runtime-only editable install: + +```bash +git clone https://github.com/MikiEEE/SmallOS.git +cd SmallOS +python3.10 -m venv .venv +source .venv/bin/activate +python -m pip install -e . +``` + +### Development install + +Install the development tools when you plan to change or validate smallOS: + +```bash +python3.10 -m venv .venv +source .venv/bin/activate +python -m pip install -e ".[dev]" +``` + +The `dev` extra installs the build, coverage, and Pyright tools used by the +repository. Run these commands from the repository root. + +## Deploy to MicroPython + +MicroPython boards do not generally consume CPython wheels. Deploy the source +package with the file-transfer or frozen-module workflow supported by your +firmware and board tooling: + +1. Install a recent firmware build for the target board. +2. Copy `SmallPackage/` to the board filesystem or freeze it into the firmware. +3. Do not import `SmallPackage.adapters` on MicroPython; those backends require + CPython threads and asyncio. +4. Copy the selected application or demo plus any configuration it loads. +5. Select `ESP32`, `PicoW`, or `build_micropython_kernel()` in the entry point. +6. Verify timer, socket, DNS, polling, Wi-Fi, and TLS behavior on the exact + firmware build before deployment. + +The board demos import `common.py`, which loads the repository-level +`smallos.config.json`. For a standalone board application, either deploy those +files alongside the demo or construct `SmallOSConfig` directly in application +code. + +### Current board-support status + +| Profile | Intended environment | Important validation | +| --- | --- | --- | +| `Unix` | CPython 3.10+ | Unit-tested in CI on Python 3.10–3.13 | +| `ESP32` | MicroPython with `network.WLAN` | Wi-Fi, polling, DNS, sockets, and TLS on target firmware | +| `PicoW` | Pico W MicroPython | Country/power settings, Wi-Fi, polling, sockets, and TLS | +| `ESP8266` | Compatible MicroPython ports | Memory limits and all network capabilities | +| `MicroPythonKernel` | Custom/constrained ports | Every capability used by the application | + +The repository does not yet claim a versioned hardware/firmware certification +matrix. Record the board model and firmware version alongside application test +results. + +## Validate the checkout + +Run the unit tests: + +```bash +python -m unittest discover -s tests -v +``` + +Run the same branch-coverage threshold used by CI: + +```bash +coverage run -m unittest discover -s tests -v +coverage report +``` + +Run static analysis and build the distributions: + +```bash +pyright +python -m build +``` + +GitHub Actions runs Pyright, unit tests on CPython 3.10 through 3.13, branch +coverage with a 60% floor, and distribution verification. Packaging begins +only after the earlier gates pass; CI installs the built wheel and smoke-tests +it outside the source checkout. + +The package includes a `py.typed` marker. Type coverage is being expanded by +subsystem, so a clean Pyright run represents the configured boundary rather +than a claim that every legacy module is strictly typed. + +## Run a demo + +After installation, verify the desktop scheduler: + +```bash +python demos/unix_demo.py +``` + +Continue with the [quick start](quick-start.md), or browse all available +[demos](demos.md). diff --git a/guide/kernels-and-micropython.md b/guide/kernels-and-micropython.md new file mode 100644 index 0000000..1395486 --- /dev/null +++ b/guide/kernels-and-micropython.md @@ -0,0 +1,61 @@ +# Kernels and MicroPython + +[Previous: Error handling](error-handling.md) · [Guide home](README.md) · +[Next: Execution adapters](execution-adapters.md) + +Kernels isolate timing, output, socket, TLS, and readiness behavior from the +scheduler. + +## Available profiles + +Desktop: + +- `Unix` + +MicroPython: + +- `MicroPythonKernel` +- `ESP32` +- `PicoW` / `RaspberryPiPicoW` +- `ESP8266` compatibility profile + +Choose a profile directly or detect one from the firmware machine string: + +```python +from SmallPackage import ESP32, PicoW, build_micropython_kernel + +kernel = ESP32(hostname="smallos-esp32") +kernel = PicoW(country="US", hostname="smallos-pico") +kernel = build_micropython_kernel() +``` + +## MicroPython startup flow + +1. Select `ESP32`, `PicoW`, or `build_micropython_kernel()`. +2. Optionally connect Wi-Fi through the kernel helper. +3. Create `SmallOS(config=...)` and attach the kernel. +4. Fork tasks and start the runtime. + +See the [ESP32](../demos/esp32_demo.py), +[Pico W](../demos/pico_w_demo.py), and +[autodetection](../demos/micropython_autodetect_demo.py) demos. + +## Portable networking boundary + +Protocol clients should build on the shared TCP/TLS kernel surface instead of +requiring protocol-specific kernel methods. + +Passive TCP consumers should: + +1. check `supports_tcp_server()` +2. pass the opaque result from `resolve_passive_address()` unchanged to both + `socket_open()` and `socket_bind()` +3. use kernel listen, accept, address-inspection, and close methods +4. check address-reuse support independently + +Some MicroPython ports can listen without exposing `SO_REUSEADDR`. The +[web application demo](../demos/web_app_demo.py) shows complete setup and +rollback without importing platform socket APIs directly. + +MicroPython support varies by firmware and board. Validate behavior on the +target hardware before depending on a kernel capability. diff --git a/guide/networking-clients.md b/guide/networking-clients.md new file mode 100644 index 0000000..332bac8 --- /dev/null +++ b/guide/networking-clients.md @@ -0,0 +1,48 @@ +# Networking Clients + +[Previous: Execution adapters](execution-adapters.md) · [Guide home](README.md) · +[Next: Shell and TCP servers](shell-and-server.md) + +smallOS includes cooperative, dependency-free helpers for common protocols: + +- `SmallStream` for raw TCP/TLS byte streams +- `SmallHTTPClient` and `SmallSSEClient` +- `SmallRedisClient` +- `SmallMQTTClient` +- `SmallWebSocketClient` + +They suspend on smallOS kernel readiness instead of owning an asyncio loop or +starting background threads. + +## HTTP example + +```python +from SmallPackage import SmallHTTPClient, SmallOS, SmallTask, Unix + + +async def fetch_status(task): + client = SmallHTTPClient(task, base_url="https://example.com") + response = await client.get("/", params={"demo": True}) + task.OS.print("{} {}\n".format(response.status_code, response.reason)) + return response.ok + + +runtime = SmallOS().setKernel(Unix()) +runtime.fork([SmallTask(2, fetch_status, name="fetch_status")]) +runtime.startOS() +``` + +HTTP currently supports common request methods, query parameters, form and +JSON bodies, TLS, and content-length, chunked, or connection-close responses. + +Redis supports RESP commands plus helpers such as `ping`, `get`, `set`, +`delete`, `publish`, and `subscribe`. MQTT 3.1.1 supports connect, disconnect, +publish, subscribe, and inbound-message acknowledgement flows at QoS 0, 1, +and 2. + +Clients accept explicit limits and transport settings. Omitted values inherit +from [`SmallOSConfig.client_defaults`](configuration.md) when the client is +attached to a task. Unix TLS supports custom CA and client certificate paths. + +For complete constructor options, protocol behavior, and examples, read the +[client reference](../SmallPackage/clients/README.md). diff --git a/guide/quick-start.md b/guide/quick-start.md new file mode 100644 index 0000000..6dcc44d --- /dev/null +++ b/guide/quick-start.md @@ -0,0 +1,57 @@ +# Quick Start + +[Previous: Installation](installation.md) · [Guide home](README.md) · +[Next: Runtime concepts](runtime-concepts.md) + +Create a task with an `async def` function, wrap it in `SmallTask`, and hand it +to a configured runtime: + +```python +from SmallPackage import SmallOS, SmallOSConfig, SmallTask, Unix + + +async def hello(task): + task.OS.print("hello from smallOS\n") + await task.sleep(0.1) + return "done" + + +config = SmallOSConfig.from_json_file("smallos.config.json") +runtime = SmallOS(config=config).setKernel(Unix()) +runtime.setErrorHandler( + lambda event: print( + "[smallOS] task failure in {} (PID {}): {}".format( + event["task_name"] or "unnamed task", + event["task_id"], + event["exception_repr"], + ) + ) +) +runtime.fork([SmallTask(2, hello, name="hello")]) +runtime.startOS() +``` + +Save the example as `hello.py` in the repository root and run: + +```bash +python hello.py +``` + +## What happens + +1. `SmallOSConfig` loads task, priority, output, watcher, and client defaults. +2. `Unix` supplies desktop timing and I/O readiness behavior. +3. `SmallTask(2, ...)` creates a priority-2 task. +4. `fork(...)` registers it with the scheduler. +5. `startOS()` runs until no runnable work or configured eternal watcher remains. +6. `task.sleep(...)` yields control without using an asyncio event loop. + +`start()` is also available as an alias for `startOS()`. + +## Next steps + +- Learn how scheduling and task-owned awaitables work in + [Runtime concepts](runtime-concepts.md). +- Tune capacity and client defaults in [Configuration](configuration.md). +- Install a production-friendly observer using + [Error handling](error-handling.md). diff --git a/guide/runtime-concepts.md b/guide/runtime-concepts.md new file mode 100644 index 0000000..91afc5b --- /dev/null +++ b/guide/runtime-concepts.md @@ -0,0 +1,68 @@ +# Runtime Concepts + +[Previous: Quick start](quick-start.md) · [Guide home](README.md) · +[Next: Task lifecycle](task-lifecycle.md) + +## Runtime model + +smallOS is a cooperative runtime: + +- tasks are `async def` coroutines wrapped in `SmallTask` +- task code awaits smallOS-owned operations such as `task.sleep(...)`, + `task.wait_signal(...)`, `task.wait_readable(...)`, and `task.join(...)` +- the scheduler steps coroutines directly and decides which priority becomes + runnable next +- kernels provide time, output, networking, and readiness primitives + +Work must yield cooperatively. CPU-heavy functions and blocking libraries +prevent the scheduler from making progress unless they are routed through an +[execution adapter](execution-adapters.md). + +## Tasks and priority + +The first `SmallTask` argument is its priority category: + +```python +runtime.fork([ + SmallTask(1, control_loop, name="control"), + SmallTask(5, telemetry_loop, name="telemetry"), +]) +``` + +Priorities start at `1` and must be lower than the configured +`priority_levels`. Lower numbers run first; tasks at the same priority retain +FIFO behavior. + +## Waiting without blocking + +Use task methods to suspend until the runtime-owned condition is ready: + +```python +async def worker(task): + REFRESH_SIGNAL = 1 + await task.sleep(0.25) + await task.wait_signal(REFRESH_SIGNAL) + await task.wait_readable(socket_object) +``` + +Readiness objects and behavior depend on the active +[kernel](kernels-and-micropython.md). + +## Child tasks and joins + +Tasks can wait for one child with `task.join(child)` or multiple children with +`task.join_all(children)`. The scheduler retains the ownership and bookkeeping +needed to resume the parent when the requested work finishes. + +See [`demos/runtime_demo.py`](../demos/runtime_demo.py) for a broader runtime +showcase. + +The [task lifecycle guide](task-lifecycle.md) covers spawning, results, +exception propagation, cancellation, signals, and watchers in detail. + +## Asyncio compatibility + +smallOS uses Python coroutine syntax, but it does not run tasks on asyncio's +event loop. An arbitrary asyncio `Task`, `Future`, or library cannot be awaited +directly from a smallOS task. Route compatible callable factories through +`AsyncioAdapter`; use `ThreadAdapter` for synchronous blocking work. diff --git a/guide/shell-and-server.md b/guide/shell-and-server.md new file mode 100644 index 0000000..c33173a --- /dev/null +++ b/guide/shell-and-server.md @@ -0,0 +1,117 @@ +# Shell and TCP Servers + +[Previous: Networking clients](networking-clients.md) · [Guide home](README.md) · +[Next: Demos](demos.md) + +## Attach an interactive shell + +`BaseShell` runs its input loop as a normal smallOS task. This keeps runtime +inspection cooperative instead of blocking the scheduler on `input()`: + +```python +from SmallPackage import SmallOS, SmallTask, Unix +from SmallPackage.shells import BaseShell + +runtime = SmallOS().setKernel(Unix()) +shell = BaseShell(prompt="app> ", allow_python=False).setOS(runtime) +runtime.shells.append(shell) + +shell_task = shell.make_task( + priority=3, + is_watcher=True, + poll_interval=0.1, +) +runtime.fork([shell_task, SmallTask(2, application, name="application")]) +runtime.start() +``` + +Use `allow_python=False` whenever shell input is not fully trusted. The Python +command evaluates arbitrary code in the process when enabled. + +### Built-in commands + +| Command | Purpose | +| --- | --- | +| `help [command]` | Show command help and aliases. | +| `ps` | List registered tasks. | +| `stat [pid]` | Show runtime or detailed task state. | +| `count` | Show task and watcher counts. | +| `children ` | List a task's known children. | +| `signal ` | Deliver an application signal. | +| `signals ` | Inspect latched signals. | +| `kill [-r]` | Cancel one task or its descendants. | +| `toggle` | Switch between shell and application output views. | +| `io [status\|show\|flush\|clear]` | Inspect buffered application output. | +| `echo ` | Write through the shell channel. | +| `python ` | Evaluate code when explicitly enabled. | +| `exit` | End the shell task, not the whole process. | + +See [`demos/shell_demo.py`](../demos/shell_demo.py) for a deterministic scripted +session and [`demos/web_app_demo.py`](../demos/web_app_demo.py) for an +interactive shell beside a server. + +## Build a passive TCP server + +Keep platform socket details behind the kernel. A portable listener follows +this order: + +```python +kernel = task.OS.kernel +if not kernel.supports_tcp_server(): + raise NotImplementedError("passive TCP is unavailable") + +address = kernel.resolve_passive_address(host, port) +listener = kernel.socket_open(address) +try: + if kernel.supports_reuse_address(): + kernel.socket_set_reuse_address(listener, True) + kernel.socket_bind(listener, address) + kernel.socket_listen(listener, backlog) + kernel.socket_setblocking(listener, False) + + while True: + try: + client, client_address = kernel.socket_accept(listener) + except Exception as exc: + retry = kernel.socket_retry_mode(exc, "accept") + if retry == "read": + await task.wait_readable(listener) + continue + if retry == "write": + await task.wait_writable(listener) + continue + raise + + try: + kernel.socket_setblocking(client, False) + task.spawn(handle_client, args=(client, client_address)) + except BaseException: + # The accept loop still owns the stream until spawn succeeds. + kernel.socket_close(client) + raise + await task.yield_now() +finally: + kernel.socket_close(listener) +``` + +The address returned by `resolve_passive_address()` is opaque: pass it +unchanged to `socket_open()` and `socket_bind()`. Treat address reuse as a +separate capability because some MicroPython ports support listening without +exposing reuse constants. + +## Connection ownership + +After accepting a client, exactly one task should own and close that socket. +If handler creation fails, close the socket in the accept path. Once the child +is registered successfully, its `finally` block should close it. This avoids +both descriptor leaks and double-close races. + +Socket calls must classify retry behavior with +`kernel.socket_retry_mode(error, operation)`. Wait for the mode it reports; +do not assume every would-block condition waits for readability. + +The web app demo implements bounded request headers, per-client handler tasks, +setup rollback, routing, metrics, shell cancellation, and listener cleanup. +It is educational code rather than a production HTTP server: add timeouts, +request-body handling, concurrency limits, authentication, and security review +for a real service. diff --git a/guide/task-lifecycle.md b/guide/task-lifecycle.md new file mode 100644 index 0000000..9fe23fd --- /dev/null +++ b/guide/task-lifecycle.md @@ -0,0 +1,152 @@ +# Task Lifecycle and Coordination + +[Previous: Runtime concepts](runtime-concepts.md) · [Guide home](README.md) · +[Next: Configuration](configuration.md) + +## Create and register tasks + +A task routine normally accepts its attached `SmallTask` as its first argument: + +```python +from SmallPackage import SmallOS, SmallTask, Unix + + +async def sensor(task, channel, interval): + while True: + reading = read_sensor(channel) + task.OS.print("channel {}: {}\n".format(channel, reading)) + await task.sleep(interval) + + +runtime = SmallOS().setKernel(Unix()) +sensor_task = SmallTask( + 2, + sensor, + name="sensor", + args=(3, 0.25), +) +pid = runtime.fork(sensor_task) +runtime.start() +``` + +`runtime.fork(task)` returns one PID. Passing a list returns the PIDs in input +order. Registration assigns the PID, attaches the runtime, and queues a ready +task; it does not execute the routine immediately. + +Useful constructor options include: + +- `name`: diagnostic name shown by the shell and failure observer +- `args`: tuple/list for positional arguments, dictionary for keyword arguments, + or one value passed as the second routine argument +- `isReady`: whether the task enters the ready queue immediately +- `isWatcher`: whether the task represents background/watch-only work + +Priorities run from `1` through `priority_levels - 1`. Lower numbers run first. + +## Spawn child tasks + +Inside a running task, use `spawn()` to create a parent/child relationship: + +```python +async def child(task, value): + await task.sleep(0.05) + return value * 2 + + +async def parent(task): + work = task.spawn(child, priority=2, name="child", args=(21,)) + answer = await task.join(work) + return answer +``` + +`spawn()` returns the child task object. If `priority` is omitted, the child +inherits the parent's priority. The older `task.fork(SmallTask(...))` wrapper +returns a PID and remains available for compatibility, but new task code should +prefer `spawn()`. + +## Join results and failures + +`await task.join(child)` returns the child's result. If the child fails or is +cancelled, its exception is raised into the waiting parent. + +`await task.join_all(children)` returns results in the caller-supplied order, +not completion order: + +```python +async def parent(task): + first = task.spawn(child, priority=4, args=(10,)) + second = task.spawn(child, priority=2, args=(20,)) + results = await task.join_all([first, second]) + return results # [20, 40] +``` + +Duplicate targets are joined once. An unknown PID is invalid. If any joined +child fails, the first observed child exception wakes the parent immediately; +other children are not automatically cancelled. + +A retained task object exposes terminal `done`, `result`, and `exception` +properties even after the runtime removes it from the live PID registry. + +## Signals + +Signals are application-defined integer slots from `0` through `31`. Use named +constants in application code: + +```python +REFRESH_SIGNAL = 3 + + +async def receiver(task): + signal_number = await task.wait_signal(REFRESH_SIGNAL) + return signal_number + + +async def sender(task, target): + status = task.sendSignal(target.getID(), REFRESH_SIGNAL) + if status != 0: + raise RuntimeError("target task is unavailable") +``` + +A signal delivered before `wait_signal()` is latched, so the later wait can +consume it immediately. `sendSignal()` returns `0` on success and `-1` for an +invalid signal, missing runtime, or unknown PID. `getSignals()` reports latched +signals and `checkSignal(sig)` consumes one without awaiting it. + +## Cooperative waiting + +The scheduler understands only smallOS-owned awaitables: + +- `await task.sleep(seconds)` resumes after a monotonic deadline +- `await task.yield_now()` voluntarily returns to the ready queue +- `await task.wait_signal(signal)` waits for an integer signal slot +- `await task.wait_readable(obj)` waits for read readiness +- `await task.wait_writable(obj)` waits for write readiness +- `await task.join(target)` and `join_all(targets)` wait for task completion +- adapter `.call(...)` methods route foreign execution through the runtime + +Awaiting an arbitrary asyncio future produces `UnsupportedAwaitableError`. + +## Cancellation + +Request scheduler-owned cancellation through the runtime: + +```python +status = runtime.cancel_task(task_or_pid, recursive=False) +``` + +The method returns `0` when the target was cancelled and `-1` when it cannot be +resolved. `recursive=True` cancels currently registered descendants first. +Cancellation is terminal and stored as `TaskCancelledError`; a parent awaiting +that task receives the same failure. + +Cancellation does not automatically cancel sibling tasks or unrelated work. +For thread and asyncio adapter limitations, see +[Execution adapters](execution-adapters.md#lifecycle-and-cancellation). + +## Watcher tasks and runtime exit + +Watcher tasks represent background facilities such as an interactive shell or +metrics loop. With the default `eternal_watchers=False`, the runtime exits when +only watchers remain. Set `eternal_watchers=True` when watcher-only work should +keep the scheduler alive, and arrange an explicit cancellation or shutdown +path. diff --git a/guide/troubleshooting.md b/guide/troubleshooting.md new file mode 100644 index 0000000..35ff0f0 --- /dev/null +++ b/guide/troubleshooting.md @@ -0,0 +1,88 @@ +# Troubleshooting and Limitations + +[Previous: Demos](demos.md) · [Guide home](README.md) · +[Next: Contributing](contributing.md) + +Install a runtime [error handler](error-handling.md) first. Without one, a +failed detached task can look like it simply stopped producing output. + +## Common problems + +### `pip install SmallPackage` installs the wrong project + +The PyPI name belongs to an unrelated maintainer. Remove that distribution and +install the verified smallOS wheel from GitHub Releases or use a source +checkout. Confirm the import path and project origin before deployment. + +### The runtime exits while a background task remains + +If every remaining task has `isWatcher=True` and `eternal_watchers` is false, +the scheduler exits intentionally. Set the configuration field to true only +when watcher-only work should keep the application alive, and provide an +explicit shutdown path. + +### `runtime.fork(...)` raises `MaxProcessError` + +The task capacity is exhausted, or the priority is invalid. Valid priorities +are `1` through `priority_levels - 1`. Increase `task_capacity`, correct the +priority, or ensure completed work is not being replaced faster than it can +finish. + +### A task stalls the entire runtime + +Cooperative tasks must yield. A long CPU loop, blocking socket call, `time.sleep`, +or synchronous SDK call prevents every smallOS task from progressing. Break CPU +work into bounded steps with `yield_now()` or route blocking libraries through +`ThreadAdapter`. + +### `UnsupportedAwaitableError` + +smallOS does not own arbitrary asyncio futures or coroutine scheduling. Use +smallOS task awaitables, or pass an async callable to `AsyncioAdapter.call()`. +Do not pass an asyncio `Task` or `Future` created on another loop. + +### I/O waiting fails with `ValueError` + +The watched object may be closed, invalid, or unsupported by the active kernel. +The runtime fails only the waiting task and reports the event to the configured +error handler. Keep socket close ownership explicit and detach/cancel waiting +tasks before closing shared descriptors. + +### An adapter is unavailable + +The kernel must support external readiness objects. `Unix` does; MicroPython +support depends on the polling backend. Also check that the adapter is open, +has not bound to a different runtime, and has free `max_pending` capacity. + +### TLS works on desktop but not on a board + +MicroPython TLS APIs and certificate behavior vary by firmware. Validate SNI, +CA loading, verification, memory use, and handshake retry behavior on the exact +board build. Do not assume desktop certificate-path options exist unchanged. + +### Cross-thread shutdown does not wake the scheduler + +Check `kernel.supports_wakeup_channel()` before creating a channel. Generic +MicroPython kernels report this unsupported. If the shutdown request originates +inside a smallOS task, apply it directly without a cross-thread channel. + +### The shell appears to hide application output + +The shell and application have separate views. Use `toggle` to switch views and +`io status`, `io show`, or `io flush` to inspect buffered output. + +## Current limitations + +- smallOS is experimental and its APIs may evolve. +- Task scheduling is cooperative; there is no preemption. +- Arbitrary asyncio libraries require `AsyncioAdapter`. +- Running Python threads cannot be forcibly stopped on task cancellation. +- MicroPython compatibility depends on board and firmware capabilities. +- PyPI publication is blocked by distribution-name ownership. +- The bundled web server is a teaching example, not a hardened HTTP stack. +- Type checking covers configured subsystems incrementally, not every legacy + module at strict settings. + +When reporting a problem, include Python or firmware version, board/OS, kernel +profile, configuration, minimal task code, full error-handler output, and +whether the problem reproduces with the closest demo.