The Observer — Artificial Life, Stage 0

A local, interactive research prototype: an interactive WebGL 3D observer over an authoritative 2D motor world, with a randomly initialized neural child that learns a small approach-and-grasp task through parent demonstrations. The world it lives in is a real three-storey house, with its walls taken directly from an architectural model.

The house

The world is a real three-storey house. It is not authored by hand: an architectural model is sliced by tools/voxelize_house.py into the tile grid the physics collides against, so the walls the child bumps into are the walls the browser draws. Both are measured from the same geometry and verified to sit in identical coordinates.

• Ground 445 walkable tiles (160 m²), first 196 (71 m²), attic 113 (41 m²), at 0.6 m per tile over a 46 × 71 grid.
• Furniture blocks. 247 tiles on the ground, 77 on the first and 12 in the attic are occupied by the model's own furnishings, extracted by measuring which parts rest on each floor slab. The child's camera shades furniture differently from walls and from the outdoors, so the policy can tell them apart.
• Stairs connect all three storeys in both directions, refusing to go below the ground floor or above the top one. The child must be standing at the flight. The elevator machine remains in the code for multi-storey buildings; a house has no lift, and the controls say so.
• Only the storey the child is on is sensed. This floor / Cutaway / Whole house chooses what is drawn, so the storeys the child is not using can be hidden. The orbit camera stays centred on the child as it moves.
• Outside is a suburban street: road, kerbs, footway, mature trees, neighbouring houses, parked and moving cars, and neighbours walking dogs. This is scenery. It is not simulated and not part of the sensor loop.

Two details in the model needed handling, and both are worth knowing because they will recur with any architectural asset:

• The doors are drawn shut. Slicing at chest height sealed every room — only about 44% of the interior was reachable. The tool slices above the door heads and then opens the thinnest wall between stranded rooms until the storey is one connected space. It opened 3 cells on the ground floor and 5 on the first; the attic needed none. Where furniture seals a route it is shifted in preference to a wall, by a cost of 1 against 50.
• Materials do not survive a naive conversion. trimesh's COLLADA reader drops them, which leaves a blank white massing model. tools/convert_model.py walks the document with pycollada instead and recovers 124 materials, 49 of them image textures.

Drag to orbit; scroll to zoom. Switch between Room, Child's view, At the window, and Above. Stairs and floor controls sit under the canvas.

Parent. Mira walks the house on a deterministic patrol, pathfinding through doorways rather than straight lines, and never leaves the building. Training and evaluation rooms keep a stationary teacher so demonstrations stay reproducible; there is a test asserting this.

• Parent lesson teaches three acoustic examples each of ball, cup, and blocks, played from local PCM speech recordings paired with a teacher-selected object category.
• Test listening evaluates three recordings with a held-out speaking rate. Test audio never enters the learned exemplar memory. These are three constrained choices with one synthetic voice, not a speech-comprehension benchmark.

The acoustic learner starts with an empty associative memory, extracts simple time-frequency features from waveforms, and matches new sound features against learned exemplars. It uses no pretrained speech recognizer or transcripts as recognition input. Object-category labels are explicit teacher supervision; this is not autonomous visual concept discovery. The existing neural motor policy and the acoustic memory are separate systems.

The parent recordings were generated using the installed macOS Samantha voice. The adult voice synthesizer is pretrained; the child's acoustic memory is not. All lesson audio is bundled locally, and no microphone is accessed.

In isolated verification, acoustic matching changed from no learned associations (0/3) to 3/3 on the three reserved recordings. A real developmental system would need open-vocabulary learning, many speakers, noisy continuous audio, visual grounding, interaction, long-term memory, and much stronger held-out evaluations. This prototype does not reproduce how babies acquire complete language.

Important separation: the 3D view is a viewer perspective. The motor policy still receives its documented 11 × 11 overhead sensor. The rendered house, articulated animation, and the street outside are not part of the neural sensor/physics loop. Replay continues to use the compact 2D historical renderer.

Model assets. No building model is committed — see Supplying a house model. The bundled Three.js licence is in web/vendor/THREE-LICENSE.txt.

Research background: Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priors.

Open the world

While the server is running, visit http://localhost:8765 and press Begin experiment.

That button starts the live room, collects demonstrations in separate training rooms, trains a candidate neural policy, evaluates it on validation and test rooms without changing its weights, and deploys it into the live world if it passes the predeclared validation threshold. The process is quick on this machine; watch the Development tests page for its results. The broader Stage 0 graduation remains locked regardless of this small benchmark's result.

Use pause/resume and 1×/2×/4× to control the world. Timeline, Memory, Development tests, and Replay are functional. Optional parent speech uses your browser's speech synthesis and may depend on the installed voice service. No external service is otherwise required by the application.

To launch again, double-click start.command in Finder, or run from this directory:

python3 server.py

Requires Python 3.10+ and NumPy. They are already available on the machine used for this build. For a new environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python server.py

The server binds only to the loopback interface. Keep its terminal/process running while using the page. Closing the browser does not stop the server. Stopping the server stops simulation time; restarting restores the latest world and weights, initially paused. This is local software, not a public deployment or background launch service.

What is implemented

• A three-storey house derived from an architectural model: 754 walkable tiles across three floors, 336 tiles of furniture, stairs, a parent who walks it, and a ball.
• A deterministic tile-based body: four translations plus grasp, with wall and furnishing collision checks. The 3D view is a visualization of this 2D physics, not a 3D physics engine.
• An 11 × 11 egocentric overhead RGB camera, a grasp-state sensor, and a fixed communication cue. The neural policy receives pixels and these channels, not object IDs or positions. The public observer can see authoritative state.
• A randomly initialized 64-hidden-unit ReLU neural policy, trained from scratch by supervised imitation with Adam. The scripted parent demonstrates shortest-path approaches. No LLM or pretrained child model.
• 1,600 procedural training rooms and 5,394 demonstrated interactions for the fixed experiment seed. Held-out validation and test each use 80 separate room seeds. Passing requires at least 80% success within 24 actions per room.
• A protocol journaled before the run; policy hashes, version, updates, compute wall time, training interactions, baseline, validation and held-out scores recorded afterward.
• An append-only SQLite journal with canonical world frames, sensor-grounded grasp memories, control events, research metadata, and policy checkpoints. Replay paginates through recorded states, 300 frames at a time.
• Visible progress, camera input, activity descriptions, and explicit simulation disclosure. No private reasoning is generated or exposed.

The user world starts untrained. Build verification used a separate database under work/, leaving the user world untouched. Verification observed 3/80 untrained validation → 80/80 learned validation; 80/80 held-out test. These results concern only this deliberately easy benchmark.

Experimental boundaries

This is a runnable foundation, not an artificial child with human-level development. There is no active outside-neighborhood simulation, school, city economy, or time-based progression. The exterior street is animated scenery. All six full Stage 0 graduation abilities remain pending or partial.

The teacher is scripted, not a pretrained conversational parent. Browser speech cues remain an observer feature. The separate acoustic association experiment receives local recorded waveforms and learns a restricted three-category matching task. The policy is feedforward and does not retrieve the stored episodic memories. It cannot demonstrate language comprehension, delayed recall, or object permanence yet.

Room size, placements, and illumination vary by split seed; the ball retains its color and shape and the furnishings remain at the perimeter. Thus the exam checks new combinations of simple configurations, not broad visual or semantic generalization. It does not yet vary object families, communication wording, complex obstacles, or realistic camera viewpoints. All channels are intentionally minimal and documented.

Training occurs off the canonical timeline, sequentially on the local CPU. The training set is regenerated deterministically; individual training transitions are not stored in the canonical journal. The candidate is frozen before scoring. The observer cannot rerun this same sealed experiment to select a better policy. Use validation for future model development and reserve fresh test partitions before comparing new architectures.

Important next research steps: richer embodied perception and manipulation; raw audio; recurrent memory retrieval; varied objects and occlusions; independent multi-skill graduation exams. Do not expand the environment merely because this reaching task passes.

Project map

• server.py: local HTTP server and simulation/training coordinator.
• alife/world.py: authoritative world, procedural curriculum partitions, body, sensors, scripted teacher.
• alife/building.py: loads the house collision grid; the seam any other building plugs into.
• alife/data/house.json: the derived grid — walls, walkable floor, furniture, stairs.
• tools/convert_model.py: COLLADA → textured GLB, preserving materials.
• tools/voxelize_house.py: model → collision grid, opening doors the model draws shut.
• alife/agent.py: neural policy, optimizer, versionable weight serialization.
• alife/research.py: training demonstration collection, fixed protocol, read-only evaluation.
• alife/store.py: persistent event journal and replay reads.
• web/: local observer, 3D scene, legacy replay renderer, bundled Three.js (MIT license in web/vendor/THREE-LICENSE.txt), and local parent recordings. web/models/ is where a building model goes; none is committed.
• alife/listening.py: supervised acoustic exemplar memory and read-only recording tests.
• tests/test_system.py: deterministic physics, split separation, read-only evaluation, learned transfer, persistence checks.
• data/life.sqlite: your saved life; keep this database and its WAL files together while the server is running. Back up after stopping the server for a simple consistent copy.

Supplying a house model

This repository does not ship a building model. The one used during development is third-party, its licence could not be established, and redistributing it would be the licensor's call rather than ours. What is committed is alife/data/house.json — the derived collision grid, a 46 x 71 occupancy map measured from that geometry rather than the geometry itself.

The simulation runs without a model: rooms, stairs, furniture and the child's sensor all work, and the page says so. Only the drawn shell is missing.

To draw a building, put a glTF binary at web/models/house.glb and regenerate the grid from it with the commands below. Any architectural model will do — the pipeline measures whatever it is given and refuses a floorplan it cannot make walkable. If your source is a SketchUp COLLADA export, tools/convert_model.py converts it and keeps the textures.

Check the licence of whatever model you use before committing it or deploying this anywhere public.

Rebuilding the world from a model

The house is derived, not authored, so a different building can replace it. These tools are for authoring only — the server itself still needs nothing but Python 3.10+ and NumPy.

python3 -m venv .venv
.venv/bin/pip install trimesh pycollada scipy Pillow
.venv/bin/python tools/convert_model.py path/to/model.dae web/models/house.glb
.venv/bin/python tools/voxelize_house.py web/models/house.glb alife/data/house.json --parts path/to/model.dae

The first command produces the textured model the browser draws. The second measures the collision grid. --parts points the measurement at the original per-part file, because the shipped model groups meshes by material for rendering speed and that hides individual furniture. Both apply the same placement, so the tiles and the drawing stay in register; the voxeliser prints what it found and refuses a floorplan it cannot make walkable.

Bumping the floorplan means bumping LAYOUT_VERSION in alife/world.py. A saved world on an older layout is rebuilt on next start, keeping its tick count, policy and journal.

Run verification:

OPENBLAS_NUM_THREADS=1 python3 -m unittest discover -s tests -v

An independent world can use another data file and port without replacing existing history:

python3 server.py --port 8766 --data data/another-life.sqlite