diff --git a/.gitignore b/.gitignore index 058ec05..ae9df15 100644 --- a/.gitignore +++ b/.gitignore @@ -1,8 +1,12 @@ # project-wise generated files run/ database/ +/database database_archive database_archive/ + +# Third-party equipment manuals -- vendor-copyrighted, not redistributable here. +pascal_manuals/ pascal/storage pascal/data old/pascal/data diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..3bc7621 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Haotong Liang and the Lumi-Lab contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index a73ea67..e14a7ed 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,15 @@ # Lumi-Lab -`lumi` — the contract-driven control stack for the PLD lab. Each piece of equipment runs -as a **node** (a process in `nodes/`) that talks to the others over RabbitMQ. What a node -can do is declared once in `src/lumi/contracts/`, and both the Python clients and the -frontend's TypeScript client are generated from that declaration. +`lumi` — the contract-driven control stack for a pulsed-laser-deposition (PLD) lab. Each +piece of equipment runs as a **node** (a process in `nodes/`) that talks to the others +over RabbitMQ. What a node can do is declared once in `src/lumi/contracts/`, and both the +Python clients and the frontend's TypeScript client are generated from that declaration. + +This is the platform behind +[*Autonomous epitaxial atomic-layer synthesis via real-time computer vision of electron +diffraction*](https://arxiv.org/abs/2602.20432): the growth driver, chamber and RHEED +nodes, and the closed-loop Bayesian-optimisation notebook here are the ones that ran the +campaigns in that paper. See [Citation](#citation) below. Nodes: @@ -316,20 +322,18 @@ host that really does talk to the chamber, set `rabbitmq.host` in your own `cfg/settings.example.toml`. The broker address is a machine-local fact, and having it in the shared template is what made `python -m lumi.mcp` reach for the lab by default. -> **The lab broker is not ready for this yet.** It still holds the pre-refactor -> messaging layer's exchanges — `CHAMBER`, `RHEED` and `STORAGE` exist there as -> non-durable **`direct`** exchanges, with live bindings from the old-style nodes. Any -> contract-era client that declares them as `topic` is refused at startup: +> **Point it at a broker only this stack uses.** Every node declares its exchanges as +> durable `topic` exchanges at startup. If the broker already carries exchanges of the +> same name with different settings — e.g. non-durable `direct` ones left by another +> messaging layer — RabbitMQ refuses the declaration: > > ``` > PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'RHEED' in vhost '/': > received 'topic' but current is 'direct' > ``` > -> This is not specific to the MCP server — it will happen to any refactored node pointed -> at that broker. Clearing it means deleting those three exchanges (they are non-durable, -> so a broker restart drops them anyway) once the old nodes are no longer using them. -> Until then, use a local broker. +> Use a dedicated vhost or a fresh broker, and delete any conflicting exchanges first +> (non-durable ones are dropped by a broker restart anyway). The tools are generated, one per `(contract, capability, op)`, named `experiment.driver.to_temperature` and so on — 49 of them today. Adding an op to @@ -403,8 +407,8 @@ travel limits). It runs entirely in-process with simulated time, so it's fast. ## Notebooks -`notebooks/` holds the two operator notebooks, ported from the v1.0 ones and written to -run against the simulator as-is: +`notebooks/` holds the two operator notebooks, written to run against the simulator +as-is: | notebook | what it does | | --- | --- | @@ -439,3 +443,24 @@ uv run lumi-codegen --check # CI gate: fails if anything on disk has drifte The contract hash printed by those commands gates backend/frontend compatibility — if it changes, the frontend needs the regenerated `lumi.ts`. See `docs/FRONTEND_MIGRATION.md` for how the browser-side API maps onto the bridge. + +## Citation + +If this software is useful in your research, please cite: + +```bibtex +@article{liang2026autonomous, + title = {Autonomous epitaxial atomic-layer synthesis via real-time computer + vision of electron diffraction}, + author = {Liang, Haotong and Sun, Yunlong and Paxson, Ryan and Lee, Chih-Yu and + Hall, Alex T. and Warecki, Zoey and Cumings, John and Koinuma, Hideomi and + Kusne, Aaron Gilad and Lippmaa, Mikk and Takeuchi, Ichiro}, + journal = {arXiv preprint arXiv:2602.20432}, + year = {2026}, + url = {https://arxiv.org/abs/2602.20432} +} +``` + +## License + +[MIT](LICENSE) © 2026 Haotong Liang and the Lumi-Lab contributors. diff --git a/notebooks/BODeposition.ipynb b/notebooks/BODeposition.ipynb index 30b1f60..cdd90eb 100644 --- a/notebooks/BODeposition.ipynb +++ b/notebooks/BODeposition.ipynb @@ -7,24 +7,13 @@ "# BO Deposition\n", "\n", "Closed-loop Bayesian optimisation over growth conditions: propose → grow →\n", - "measure → refit, one substrate position per iteration. Ported from\n", - "`UMDAutonomousExperiment.ipynb`.\n", + "measure → refit, one substrate position per iteration. This is the loop from\n", + "[arXiv:2602.20432](https://arxiv.org/abs/2602.20432) -- the GP, the UCB /\n", + "max-uncertainty acquisition functions and the convergence check live in `lumi.opt`.\n", "\n", - "The loop itself is unchanged -- same GP, same UCB/max-uncertainty acquisition\n", - "functions, same convergence check, ported in `lumi.opt` from the code that ran the real\n", - "campaigns. What changed is where the campaign's memory lives:\n", - "\n", - "| v1.0 | now |\n", - "| --- | --- |\n", - "| `GPManager` + `gp_db/.csv` | `GrowthCampaign`, reading `list_measurements` |\n", - "| `collector` pickle, `local_experiment_counter` | `list_samples` -- positions are rows |\n", - "| `substrate.has_lsmo` monkeypatched on | the derived layer stack |\n", - "| metric kept beside the notebook | a `measurement` row on the sample |\n", - "| `PixelExperimentManager.perform_experiment` | `recipes.perform_pixel_deposition` |\n", - "\n", - "That the training set comes from the growth database and not a local CSV is the\n", - "substantive change: a campaign can now be resumed from a different machine, and the GP\n", - "is fitted on what the lab actually recorded.\n", + "The campaign's memory is the growth database, not a local CSV: `GrowthCampaign` reads\n", + "its training set back through `list_measurements` each iteration, so a campaign can be\n", + "resumed from a different machine and the GP is fitted on what the lab actually recorded.\n", "\n", "**Runs against the simulator as written:**\n", "\n", @@ -81,7 +70,7 @@ "metadata": {}, "source": [ "HOST = \"localhost\"\n", - "ACTOR = \"hliang16\"\n", + "ACTOR = \"operator\"\n", "DRYRUN = True # True: the loop runs, the laser does not\n", "\n", "exp = await ExperimentSession.open(host=HOST, actor=ACTOR)\n", @@ -109,7 +98,7 @@ "cell_type": "code", "metadata": {}, "source": [ - "PROJECT = \"UMD_AI_HZO_BO\"\n", + "PROJECT = \"hzo_bo_demo\"\n", "TARGET_SLOT = \"D\" # the functional layer's carousel slot\n", "N_RANDOM = 2 # random seed points before the GP takes over\n", "NUM_PULSE = 120 # real: 500 (~9 nm of HZO)\n", @@ -135,10 +124,10 @@ "source": [ "## The optimiser\n", "\n", - "Two acquisition functions, exactly as the production campaign ran them: UCB with\n", - "`beta=3` to exploit, and maximum-uncertainty to break out when UCB converges on a local\n", - "maximum. The constant mean at 0.5 and the length-scale interval are the values those\n", - "runs used -- a shorter length scale under-fits and yields no useful uncertainty.\n" + "Two acquisition functions: UCB with `beta=3` to exploit, and maximum-uncertainty to\n", + "break out when UCB converges on a local maximum. The constant mean at 0.5 and the\n", + "length-scale interval are the values the published campaigns used -- a shorter length\n", + "scale under-fits and yields no useful uncertainty.\n" ] }, { @@ -183,8 +172,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "A band of one axis can be excluded -- the production campaign ruled out a pressure\n", - "window the chamber could not hold stably.\n" + "A band of one axis can be excluded -- e.g. a pressure window the chamber cannot hold\n", + "stably.\n" ] }, { @@ -288,9 +277,8 @@ " \"\"\"Block until the chamber's motors report free.\n", "\n", " `to_current_pixel` moves the mask and the RHEED gun, and refuses outright while a\n", - " previous move is still running. The production notebook covered this with a manual\n", - " \"check motor free\" gate before each growth; an unattended loop has to wait for it\n", - " itself.\n", + " previous move is still running. An unattended loop has to wait for it rather than\n", + " relying on a person to check \"motor free\" before each growth.\n", " \"\"\"\n", " deadline = asyncio.get_running_loop().time() + timeout\n", " while asyncio.get_running_loop().time() < deadline:\n", @@ -319,24 +307,22 @@ "source": [ "## Scoring a growth\n", "\n", - "In the lab this is `analyze_rheed_video` from the RHEED analysis stack, which needs the\n", - "detection host's `rhana`/`mmdet` install and a real recorded HDF5. It returns growth,\n", - "speed, quality and roughness components plus the combined `metric`.\n", + "In the lab this is a RHEED-video analysis that reads the recorded HDF5 and returns\n", + "growth, speed, quality and roughness components plus the combined `metric`. It needs the\n", + "detection host's model stack (`rhana` / `mmdet`) and a real recording.\n", "\n", "Against the simulator there is no film to score, so the fallback below stands in a\n", - "smooth synthetic landscape -- enough to watch the loop converge on something. Swap\n", - "`score_growth` for the real analysis on the detection host.\n" + "smooth synthetic landscape -- enough to watch the loop converge on something. Point\n", + "`score_growth` at the real analysis on the detection host.\n" ] }, { "cell_type": "code", "metadata": {}, "source": [ - "try:\n", - " from src.analysis import analyze_rheed_video # the lab's RHEED analysis stack\n", - " HAVE_ANALYSIS = True\n", - "except ImportError:\n", - " HAVE_ANALYSIS = False\n", + "# Set this True on the detection host, where the RHEED-video analysis is importable,\n", + "# and fill in the call below.\n", + "HAVE_ANALYSIS = False\n", "\n", "print(\"real RHEED analysis available:\", HAVE_ANALYSIS)\n", "\n", @@ -344,7 +330,7 @@ "async def score_growth(storage_name: str, conditions: pd.Series) -> dict:\n", " \"\"\"Return the metric dict for a finished growth.\n", "\n", - " Replace the fallback with analyze_rheed_video(...) on the detection host:\n", + " On the detection host, replace the fallback with the real analysis, e.g.:\n", "\n", " metrics, _ = await to_async(\n", " analyze_rheed_video,\n", @@ -355,8 +341,8 @@ " \"\"\"\n", " if HAVE_ANALYSIS:\n", " raise NotImplementedError(\n", - " \"wire analyze_rheed_video in here -- it needs the recorded HDF5 and the \"\n", - " \"crop/periodicity setup from the production notebook\"\n", + " \"wire the RHEED-video analysis in here -- it needs the recorded HDF5 and \"\n", + " \"the crop / periodicity setup for your chamber\"\n", " )\n", "\n", " # Synthetic stand-in: a smooth optimum inside the search box, plus noise.\n", @@ -379,11 +365,11 @@ "Propose → grow → score → record → refit, until the substrate runs\n", "out of positions or the GP converges.\n", "\n", - "Two differences from the v1.0 loop worth pointing at. `campaign.refresh(exp.driver)`\n", - "reloads the training set from the growth database each iteration, so the GP sees every\n", - "measurement the lab has -- including ones added from another machine, or from a\n", - "re-analysis. And there is no `collector` to keep in step: the position that was grown,\n", - "what went on it and what it scored are all rows the system wrote itself.\n" + "One thing worth pointing at: `campaign.refresh(exp.driver)` reloads the training set\n", + "from the growth database each iteration, so the GP sees every measurement the lab has --\n", + "including ones added from another machine, or from a re-analysis. There is no separate\n", + "bookkeeping to keep in step: the position that was grown, what went on it and what it\n", + "scored are all rows the system wrote itself.\n" ] }, { @@ -407,8 +393,7 @@ " print(f\"\\nconverged after {iteration} growths -- stopping\")\n", " break\n", " if is_converged:\n", - " # A local maximum: one round of pure exploration to break out. Same\n", - " # escape the production campaign used.\n", + " # A local maximum: one round of pure exploration to break out.\n", " conditions = campaign.propose(\"exploration\")\n", " mode = \"exploration (escaping a local maximum)\"\n", " else:\n", diff --git a/notebooks/README.md b/notebooks/README.md index 2db192b..d8c8ef9 100644 --- a/notebooks/README.md +++ b/notebooks/README.md @@ -1,14 +1,16 @@ # Notebooks -Ported from the v1.0 notebooks in `~/HZO_PLD/OpCode/`, onto the current contract-driven -API. Both are written **sim-first**: they run end to end against +Two operator notebooks, written **sim-first**: they run end to end against `scripts/start_simulation.sh` as written, with the real-hardware numbers kept in one clearly marked config cell each. -| notebook | what it does | ported from | -| --- | --- | --- | -| `SingleDeposition.ipynb` | a layered growth on one position — bottom electrode → interface → functional layer | `UMDSingleDepoExperiment.ipynb` | -| `BODeposition.ipynb` | closed-loop Bayesian optimisation over growth conditions, one position per iteration | `UMDAutonomousExperiment.ipynb` | +| notebook | what it does | +| --- | --- | +| `SingleDeposition.ipynb` | a layered growth on one position — bottom electrode → interface → functional layer | +| `BODeposition.ipynb` | closed-loop Bayesian optimisation over growth conditions, one position per iteration | + +`BODeposition.ipynb` runs the loop from +[arXiv:2602.20432](https://arxiv.org/abs/2602.20432). ## Running them @@ -36,8 +38,8 @@ uv sync --extra opt # torch, gpytorch ``` To switch either notebook to real hardware, edit `HOST`, set `DRYRUN = False`, and -replace the values in the marked parameters cell — the production numbers are in the -comment on each line. +replace the values in the marked parameters cell — representative real numbers are in +the comment on each line. ## Editing them @@ -66,14 +68,14 @@ layers grown, every step journaled with its actor, the derived layer stack read `BODeposition.ipynb` has also been executed end to end: 2 random-seed growths then 5 GP-proposed ones, all positions on the substrate used, no exception. That confirms the loop's mechanics (bookkeeping, gates, journaling, GP wiring) but **not** its -optimisation behaviour under `DRYRUN = True` -- `to_temperature` is skipped outright in +optimisation behaviour under `DRYRUN = True` — `to_temperature` is skipped outright in dryrun and pressure is always a printed manual instruction with nothing to act on it in the simulator, so the conditions the GP trains on don't match what was proposed. See `docs/SAMPLE_TRACKING.md` for the detail and the remaining to-do. -## `analyze_rheed_video` +## Scoring a growth -The BO notebook's `score_growth` is where the real RHEED analysis goes. It needs the -detection host's `rhana`/`mmdet` install and a recorded HDF5, neither of which the -simulator has, so against the simulator it falls back to a synthetic landscape and -says so in its output. Swap it for `analyze_rheed_video(...)` on the detection host. +The BO notebook's `score_growth` is where the real RHEED-video analysis goes. It needs +the detection host's `rhana`/`mmdet` install and a recorded HDF5, neither of which the +simulator has, so against the simulator it falls back to a synthetic landscape and says +so in its output. Point it at the real analysis on the detection host. diff --git a/notebooks/SingleDeposition.ipynb b/notebooks/SingleDeposition.ipynb index 326e7c0..b84e398 100644 --- a/notebooks/SingleDeposition.ipynb +++ b/notebooks/SingleDeposition.ipynb @@ -7,19 +7,10 @@ "# Single Deposition\n", "\n", "A layered growth on one substrate position: **bottom electrode → interface →\n", - "functional layer**. Ported from `UMDSingleDepoExperiment.ipynb`, which drove the real\n", - "HZO stack (SRO / LSMO / HZO) against the v1.0 API.\n", - "\n", - "What changed in the port:\n", - "\n", - "| v1.0 | now |\n", - "| --- | --- |\n", - "| `MQCommunication()` + eleven clients | `ExperimentSession` |\n", - "| `SingleDepoExperimentManager.perform_experiment` | `recipes.perform_single_deposition` |\n", - "| `ainput(\"...\")` blocking the kernel | `input_provider` / `value_provider` hooks |\n", - "| `src.db.GrowthDB` opened locally | the experiment node owns it; reached over the contract |\n", - "| `collector` pickle + `local_experiment_counter` | `list_samples` / `sample_history` |\n", - "| conditions typed into a dict | recorded automatically as `step` rows |\n", + "functional layer**. One `ExperimentSession` opens the bus; `recipes.perform_single_deposition`\n", + "runs each layer end to end, pausing at every point a person has to do or read something\n", + "physical; and every world-changing op is recorded as a `step` row, so the sample's\n", + "history is written by the system rather than kept in a notebook variable.\n", "\n", "**This notebook runs against the simulator as written.** Bring the stack up first:\n", "\n", @@ -64,8 +55,7 @@ "## Connect\n", "\n", "`ExperimentSession` spans the three exchanges a growth needs (EXPERIMENT + RHEED +\n", - "CHAMBER) and owns the connection. It replaces the eleven separate clients the v1.0\n", - "notebook opened by hand.\n", + "CHAMBER) and owns the connection.\n", "\n", "`actor` is who to credit in the step journal. Every op you run below is recorded\n", "against this name, so the history says *who* ramped the chamber rather than just that\n", @@ -76,8 +66,8 @@ "cell_type": "code", "metadata": {}, "source": [ - "HOST = \"localhost\" # the lab broker's address on the instrument machine\n", - "ACTOR = \"hliang16\" # your name, for the step journal\n", + "HOST = \"localhost\" # the broker's address; the instrument machine in the lab\n", + "ACTOR = \"operator\" # your name, for the step journal\n", "DRYRUN = True # True: bookkeeping and gates, no laser and no ramp\n", "\n", "exp = await ExperimentSession.open(host=HOST, actor=ACTOR)\n", @@ -115,8 +105,7 @@ "source": [ "## Look at the RHEED pattern\n", "\n", - "The v1.0 notebook reached for `comm.camera_client.get_image()`. The session holds the\n", - "RHEED camera directly, for exactly this.\n" + "The session holds the RHEED camera directly, for exactly this.\n" ] }, { @@ -142,7 +131,7 @@ "## Growth parameters\n", "\n", "**← This is the cell to edit for a real run.** The values below are the\n", - "simulator's; the real HZO stack's numbers from the production notebook are in the\n", + "simulator's; representative real numbers for an HZO stack (SRO / LSMO / HZO) are in the\n", "comment on each line.\n" ] }, @@ -183,14 +172,14 @@ "## Answering the recipe's prompts\n", "\n", "`perform_single_deposition` pauses at every point a person has to do or read something\n", - "physical. The v1.0 notebook used `ainput`, which blocks the kernel on stdin -- awkward\n", - "in Jupyter and impossible to leave unattended.\n", - "\n", - "The recipe takes two hooks instead. Below they answer themselves so the notebook runs\n", - "end to end against the simulator with nobody at the keyboard. A real run needs an\n", - "actual person reading the laser power meter and the RHEED screen, so it gets\n", - "`recipes.default_input_provider` / `default_value_provider` -- the same terminal-input\n", - "functions everyone in the lab actually uses -- selected the moment `DRYRUN` is off.\n" + "physical. It takes two hooks for this -- an `input_provider` for \"do X now\" messages and\n", + "a `value_provider` for questions that need an answer back.\n", + "\n", + "Below they answer themselves so the notebook runs end to end against the simulator with\n", + "nobody at the keyboard. A real run needs an actual person reading the laser power meter\n", + "and the RHEED screen, so it gets `recipes.default_input_provider` /\n", + "`default_value_provider` -- terminal-input functions -- selected the moment `DRYRUN` is\n", + "off.\n" ] }, { @@ -225,9 +214,9 @@ "source": [ "## Register the project and substrate\n", "\n", - "`register_substrate` now also materialises one **sample** row per growable position,\n", - "so \"which positions are spent\" is a query rather than something the notebook has to\n", - "remember. The v1.0 notebook kept that in a pickle.\n" + "`register_substrate` also materialises one **sample** row per growable position, so\n", + "\"which positions are spent\" is a query rather than something the notebook has to\n", + "remember.\n" ] }, { @@ -260,8 +249,7 @@ "### Resuming instead\n", "\n", "If the node restarted mid-campaign, resume rather than register -- `resume_substrate`\n", - "now restores which positions are already grown on, so it will not hand you a spent\n", - "one. (It used to rewind to position 0 and deposit on top of an existing film.)\n", + "restores which positions are already grown on, so it will not hand you a spent one.\n", "\n", "```python\n", "substrate = await exp.driver.resume_substrate(ResumeSubstrate(substrate_id=))\n", @@ -302,9 +290,8 @@ "source": [ "## Grow the stack\n", "\n", - "One `perform_single_deposition` per layer. The v1.0 notebook had this copy-pasted\n", - "three times with a `collector` dict assembled by hand after each; the loop below is the\n", - "same sequence, and the bookkeeping it was doing manually is now the journal's job.\n", + "One `perform_single_deposition` per layer, in a loop -- the per-layer bookkeeping is the\n", + "journal's job, not something to assemble by hand after each call.\n", "\n", "`finish_substrate=False` on every layer but the last: all three go on the *same*\n", "position, and retiring it after the first would leave nothing to grow on.\n" diff --git a/notebooks/_bo_deposition.py b/notebooks/_bo_deposition.py index 1011ecf..15e5fd7 100644 --- a/notebooks/_bo_deposition.py +++ b/notebooks/_bo_deposition.py @@ -12,24 +12,13 @@ def cells() -> list[tuple[str, str]]: # BO Deposition Closed-loop Bayesian optimisation over growth conditions: propose → grow → -measure → refit, one substrate position per iteration. Ported from -`UMDAutonomousExperiment.ipynb`. +measure → refit, one substrate position per iteration. This is the loop from +[arXiv:2602.20432](https://arxiv.org/abs/2602.20432) -- the GP, the UCB / +max-uncertainty acquisition functions and the convergence check live in `lumi.opt`. -The loop itself is unchanged -- same GP, same UCB/max-uncertainty acquisition -functions, same convergence check, ported in `lumi.opt` from the code that ran the real -campaigns. What changed is where the campaign's memory lives: - -| v1.0 | now | -| --- | --- | -| `GPManager` + `gp_db/.csv` | `GrowthCampaign`, reading `list_measurements` | -| `collector` pickle, `local_experiment_counter` | `list_samples` -- positions are rows | -| `substrate.has_lsmo` monkeypatched on | the derived layer stack | -| metric kept beside the notebook | a `measurement` row on the sample | -| `PixelExperimentManager.perform_experiment` | `recipes.perform_pixel_deposition` | - -That the training set comes from the growth database and not a local CSV is the -substantive change: a campaign can now be resumed from a different machine, and the GP -is fitted on what the lab actually recorded. +The campaign's memory is the growth database, not a local CSV: `GrowthCampaign` reads +its training set back through `list_measurements` each iteration, so a campaign can be +resumed from a different machine and the GP is fitted on what the lab actually recorded. **Runs against the simulator as written:** @@ -72,7 +61,7 @@ def cells() -> list[tuple[str, str]]: """), (CODE, """ HOST = "localhost" -ACTOR = "hliang16" +ACTOR = "operator" DRYRUN = True # True: the loop runs, the laser does not exp = await ExperimentSession.open(host=HOST, actor=ACTOR) @@ -90,7 +79,7 @@ def cells() -> list[tuple[str, str]]: gives a length scale dominated by the top of the range. """), (CODE, """ -PROJECT = "UMD_AI_HZO_BO" +PROJECT = "hzo_bo_demo" TARGET_SLOT = "D" # the functional layer's carousel slot N_RANDOM = 2 # random seed points before the GP takes over NUM_PULSE = 120 # real: 500 (~9 nm of HZO) @@ -110,10 +99,10 @@ def cells() -> list[tuple[str, str]]: (MD, """ ## The optimiser -Two acquisition functions, exactly as the production campaign ran them: UCB with -`beta=3` to exploit, and maximum-uncertainty to break out when UCB converges on a local -maximum. The constant mean at 0.5 and the length-scale interval are the values those -runs used -- a shorter length scale under-fits and yields no useful uncertainty. +Two acquisition functions: UCB with `beta=3` to exploit, and maximum-uncertainty to +break out when UCB converges on a local maximum. The constant mean at 0.5 and the +length-scale interval are the values the published campaigns used -- a shorter length +scale under-fits and yields no useful uncertainty. """), (CODE, """ import gpytorch @@ -148,8 +137,8 @@ def cells() -> list[tuple[str, str]]: print(f"search grid: {len(campaign._test_x_raw)} points over {campaign.x_columns}") """), (MD, """ -A band of one axis can be excluded -- the production campaign ruled out a pressure -window the chamber could not hold stably. +A band of one axis can be excluded -- e.g. a pressure window the chamber cannot hold +stably. """), (CODE, """ removed = campaign.forbid("pressure", 1e-4, 2e-3) @@ -219,9 +208,8 @@ async def wait_for_motor(timeout: float = 120.0) -> None: \"\"\"Block until the chamber's motors report free. `to_current_pixel` moves the mask and the RHEED gun, and refuses outright while a - previous move is still running. The production notebook covered this with a manual - "check motor free" gate before each growth; an unattended loop has to wait for it - itself. + previous move is still running. An unattended loop has to wait for it rather than + relying on a person to check "motor free" before each growth. \"\"\" deadline = asyncio.get_running_loop().time() + timeout while asyncio.get_running_loop().time() < deadline: @@ -244,20 +232,18 @@ def random_conditions() -> pd.Series: (MD, """ ## Scoring a growth -In the lab this is `analyze_rheed_video` from the RHEED analysis stack, which needs the -detection host's `rhana`/`mmdet` install and a real recorded HDF5. It returns growth, -speed, quality and roughness components plus the combined `metric`. +In the lab this is a RHEED-video analysis that reads the recorded HDF5 and returns +growth, speed, quality and roughness components plus the combined `metric`. It needs the +detection host's model stack (`rhana` / `mmdet`) and a real recording. Against the simulator there is no film to score, so the fallback below stands in a -smooth synthetic landscape -- enough to watch the loop converge on something. Swap -`score_growth` for the real analysis on the detection host. +smooth synthetic landscape -- enough to watch the loop converge on something. Point +`score_growth` at the real analysis on the detection host. """), (CODE, """ -try: - from src.analysis import analyze_rheed_video # the lab's RHEED analysis stack - HAVE_ANALYSIS = True -except ImportError: - HAVE_ANALYSIS = False +# Set this True on the detection host, where the RHEED-video analysis is importable, +# and fill in the call below. +HAVE_ANALYSIS = False print("real RHEED analysis available:", HAVE_ANALYSIS) @@ -265,7 +251,7 @@ def random_conditions() -> pd.Series: async def score_growth(storage_name: str, conditions: pd.Series) -> dict: \"\"\"Return the metric dict for a finished growth. - Replace the fallback with analyze_rheed_video(...) on the detection host: + On the detection host, replace the fallback with the real analysis, e.g.: metrics, _ = await to_async( analyze_rheed_video, @@ -276,8 +262,8 @@ async def score_growth(storage_name: str, conditions: pd.Series) -> dict: \"\"\" if HAVE_ANALYSIS: raise NotImplementedError( - "wire analyze_rheed_video in here -- it needs the recorded HDF5 and the " - "crop/periodicity setup from the production notebook" + "wire the RHEED-video analysis in here -- it needs the recorded HDF5 and " + "the crop / periodicity setup for your chamber" ) # Synthetic stand-in: a smooth optimum inside the search box, plus noise. @@ -294,11 +280,11 @@ async def score_growth(storage_name: str, conditions: pd.Series) -> dict: Propose → grow → score → record → refit, until the substrate runs out of positions or the GP converges. -Two differences from the v1.0 loop worth pointing at. `campaign.refresh(exp.driver)` -reloads the training set from the growth database each iteration, so the GP sees every -measurement the lab has -- including ones added from another machine, or from a -re-analysis. And there is no `collector` to keep in step: the position that was grown, -what went on it and what it scored are all rows the system wrote itself. +One thing worth pointing at: `campaign.refresh(exp.driver)` reloads the training set +from the growth database each iteration, so the GP sees every measurement the lab has -- +including ones added from another machine, or from a re-analysis. There is no separate +bookkeeping to keep in step: the position that was grown, what went on it and what it +scored are all rows the system wrote itself. """), (CODE, """ history = [] @@ -318,8 +304,7 @@ async def score_growth(storage_name: str, conditions: pd.Series) -> dict: print(f"\\nconverged after {iteration} growths -- stopping") break if is_converged: - # A local maximum: one round of pure exploration to break out. Same - # escape the production campaign used. + # A local maximum: one round of pure exploration to break out. conditions = campaign.propose("exploration") mode = "exploration (escaping a local maximum)" else: diff --git a/notebooks/_single_deposition.py b/notebooks/_single_deposition.py index 4f6a6d1..e21625d 100644 --- a/notebooks/_single_deposition.py +++ b/notebooks/_single_deposition.py @@ -12,19 +12,10 @@ def cells() -> list[tuple[str, str]]: # Single Deposition A layered growth on one substrate position: **bottom electrode → interface → -functional layer**. Ported from `UMDSingleDepoExperiment.ipynb`, which drove the real -HZO stack (SRO / LSMO / HZO) against the v1.0 API. - -What changed in the port: - -| v1.0 | now | -| --- | --- | -| `MQCommunication()` + eleven clients | `ExperimentSession` | -| `SingleDepoExperimentManager.perform_experiment` | `recipes.perform_single_deposition` | -| `ainput("...")` blocking the kernel | `input_provider` / `value_provider` hooks | -| `src.db.GrowthDB` opened locally | the experiment node owns it; reached over the contract | -| `collector` pickle + `local_experiment_counter` | `list_samples` / `sample_history` | -| conditions typed into a dict | recorded automatically as `step` rows | +functional layer**. One `ExperimentSession` opens the bus; `recipes.perform_single_deposition` +runs each layer end to end, pausing at every point a person has to do or read something +physical; and every world-changing op is recorded as a `step` row, so the sample's +history is written by the system rather than kept in a notebook variable. **This notebook runs against the simulator as written.** Bring the stack up first: @@ -59,16 +50,15 @@ def cells() -> list[tuple[str, str]]: ## Connect `ExperimentSession` spans the three exchanges a growth needs (EXPERIMENT + RHEED + -CHAMBER) and owns the connection. It replaces the eleven separate clients the v1.0 -notebook opened by hand. +CHAMBER) and owns the connection. `actor` is who to credit in the step journal. Every op you run below is recorded against this name, so the history says *who* ramped the chamber rather than just that a notebook did. """), (CODE, """ -HOST = "localhost" # the lab broker's address on the instrument machine -ACTOR = "hliang16" # your name, for the step journal +HOST = "localhost" # the broker's address; the instrument machine in the lab +ACTOR = "operator" # your name, for the step journal DRYRUN = True # True: bookkeeping and gates, no laser and no ramp exp = await ExperimentSession.open(host=HOST, actor=ACTOR) @@ -90,8 +80,7 @@ def cells() -> list[tuple[str, str]]: (MD, """ ## Look at the RHEED pattern -The v1.0 notebook reached for `comm.camera_client.get_image()`. The session holds the -RHEED camera directly, for exactly this. +The session holds the RHEED camera directly, for exactly this. """), (CODE, """ # NB: (meta, frame), not (frame, meta) -- the array never passes through JSON, @@ -107,7 +96,7 @@ def cells() -> list[tuple[str, str]]: ## Growth parameters **← This is the cell to edit for a real run.** The values below are the -simulator's; the real HZO stack's numbers from the production notebook are in the +simulator's; representative real numbers for an HZO stack (SRO / LSMO / HZO) are in the comment on each line. """), (CODE, """ @@ -138,14 +127,14 @@ def cells() -> list[tuple[str, str]]: ## Answering the recipe's prompts `perform_single_deposition` pauses at every point a person has to do or read something -physical. The v1.0 notebook used `ainput`, which blocks the kernel on stdin -- awkward -in Jupyter and impossible to leave unattended. - -The recipe takes two hooks instead. Below they answer themselves so the notebook runs -end to end against the simulator with nobody at the keyboard. A real run needs an -actual person reading the laser power meter and the RHEED screen, so it gets -`recipes.default_input_provider` / `default_value_provider` -- the same terminal-input -functions everyone in the lab actually uses -- selected the moment `DRYRUN` is off. +physical. It takes two hooks for this -- an `input_provider` for "do X now" messages and +a `value_provider` for questions that need an answer back. + +Below they answer themselves so the notebook runs end to end against the simulator with +nobody at the keyboard. A real run needs an actual person reading the laser power meter +and the RHEED screen, so it gets `recipes.default_input_provider` / +`default_value_provider` -- terminal-input functions -- selected the moment `DRYRUN` is +off. """), (CODE, """ async def auto_input(message: str) -> None: @@ -170,9 +159,9 @@ async def auto_value(prompt: str) -> str: (MD, """ ## Register the project and substrate -`register_substrate` now also materialises one **sample** row per growable position, -so "which positions are spent" is a query rather than something the notebook has to -remember. The v1.0 notebook kept that in a pickle. +`register_substrate` also materialises one **sample** row per growable position, so +"which positions are spent" is a query rather than something the notebook has to +remember. """), (CODE, """ project = await exp.driver.register_project(RegisterProject( @@ -195,8 +184,7 @@ async def auto_value(prompt: str) -> str: ### Resuming instead If the node restarted mid-campaign, resume rather than register -- `resume_substrate` -now restores which positions are already grown on, so it will not hand you a spent -one. (It used to rewind to position 0 and deposit on top of an existing film.) +restores which positions are already grown on, so it will not hand you a spent one. ```python substrate = await exp.driver.resume_substrate(ResumeSubstrate(substrate_id=)) @@ -223,9 +211,8 @@ async def auto_value(prompt: str) -> str: (MD, """ ## Grow the stack -One `perform_single_deposition` per layer. The v1.0 notebook had this copy-pasted -three times with a `collector` dict assembled by hand after each; the loop below is the -same sequence, and the bookkeeping it was doing manually is now the journal's job. +One `perform_single_deposition` per layer, in a loop -- the per-layer bookkeeping is the +journal's job, not something to assemble by hand after each call. `finish_substrate=False` on every layer but the last: all three go on the *same* position, and retiring it after the first would leave nothing to grow on. diff --git a/pyproject.toml b/pyproject.toml index f3a36c6..8f13114 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -3,6 +3,8 @@ name = "lumi" version = "0.1.0" description = "Autonomous PLD and RHEED equipment control" readme = "README.md" +license = "MIT" +license-files = ["LICENSE"] # Floor of 3.11: real code uses datetime.UTC and enum.StrEnum (both added in 3.11). # Capped below 3.12: mmdet hard-requires mmcv<2.2.0, but OpenMMLab only publishes # prebuilt mmcv<2.2.0 wheels up to cp311 -- see the `detection` extra below and diff --git a/scripts/start_server_host.sh b/scripts/start_server_host.sh index 60fffa6..c4fe359 100755 --- a/scripts/start_server_host.sh +++ b/scripts/start_server_host.sh @@ -249,7 +249,7 @@ if [[ -z "$RESOLVED_ROOT" ]]; then fail "could not resolve the HDF5 output directory" elif [[ ! -d "$RESOLVED_ROOT" ]]; then fail "HDF5 root does not exist: $RESOLVED_ROOT" - echo " The tracked 'database' symlink points at the lab share; mount it," >&2 + echo " Create a 'database' symlink (or dir) pointing at your storage share," >&2 echo " or pass --root . Recordings are lost silently without it." >&2 elif [[ ! -w "$RESOLVED_ROOT" ]]; then fail "HDF5 root is not writable: $RESOLVED_ROOT"