Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ go-sdk/
├── instrumentation/ # tracing + metrics SPIs, no-op defaults, policies
├── config/ # layered override→env→default settings
├── serde/ # Marshaler/Unmarshaler seam + JSON default + Tristate
├── jsonl/ # NDJSON / JSON Lines stream decoder
├── sse/ # Server-Sent Events (WHATWG) parser
├── webhook/ # inbound signature verification (HMAC + timestamp)
├── formdata/ # multipart/form-data body builder
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ standard library.
| [`config`](./config) | Layered override → environment → default settings resolver; non-failing typed getters. |
| [`serde`](./serde) | Serialization seam (Marshaler/Unmarshaler) with a JSON default, plus Tristate for PATCH payloads. |
| [`sse`](./sse) | Server-Sent Events (text/event-stream) WHATWG parser + reconnecting Stream (Last-Event-ID replay). |
| [`jsonl`](./jsonl) | JSON Lines / NDJSON streaming decoder (`iter.Seq2`). |
| [`webhook`](./webhook) | Inbound webhook signature verification (constant-time HMAC + timestamp tolerance). |
| [`formdata`](./formdata) | Multipart/form-data request body builder (replayable; file uploads). |
| root [`dexpace`](.) | Umbrella `Client` wiring the default policy stack. |
Expand Down
3 changes: 3 additions & 0 deletions doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@
// JSON default) and Tristate for JSON PATCH payloads; httperr.ResponseError.DecodeInto
// decodes an error body into a typed value.
//
// The jsonl package decodes streaming JSON Lines (NDJSON) responses into typed
// values.
//
// The sse package parses Server-Sent Events (text/event-stream) into a
// range-over-func iterator of events, with a reconnecting Stream that replays the
// Last-Event-ID.
Expand Down
289 changes: 289 additions & 0 deletions docs/superpowers/plans/2026-06-16-jsonl.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,289 @@
# JSONL / NDJSON Decoder Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add a `jsonl` package with `Decode[T](r io.Reader) iter.Seq2[T, error]` that streams successive JSON values from a reader (NDJSON / JSON Lines) as typed values.

**Architecture:** A `json.Decoder` reads successive values from `r`; the iterator yields each decoded `T`, ends cleanly at `io.EOF`, and surfaces any other decode error once before stopping.

**Tech Stack:** Go 1.26+, standard library only (`encoding/json`, `errors`, `io`, `iter`). Zero third-party dependencies.

**Conventions every task must follow:**
- MIT license header on every `.go` file before the `package` clause.
- Import groups: stdlib only here.
- Tests use `t.Parallel()`; table-driven where natural; stdlib-only.
- Tools: Go 1.26.3; `gofumpt`/`golangci-lint` NOT installed — use `gofmt`, `go vet`, `go test -race`.
- Run commands from the repo root `/Users/omar/dexpace/go-sdk`.

---

## File Structure

| Path | Responsibility |
|---|---|
| `jsonl/doc.go` (new) | package comment |
| `jsonl/jsonl.go` (new) | `Decode` |
| `jsonl/jsonl_test.go` (new) | NDJSON / scalar / empty / error / break tests |
| `doc.go`, `README.md`, `CLAUDE.md` (modify) | document; add the package |

---

## Task 1: the `jsonl` package

**Files:**
- Create: `jsonl/doc.go`, `jsonl/jsonl.go`, `jsonl/jsonl_test.go`

- [ ] **Step 1: Write the failing tests**

```go
// jsonl/jsonl_test.go
// Copyright (c) 2026 dexpace and Omar Aljarrah.
// Licensed under the MIT License. See LICENSE in the repository root for details.

package jsonl_test

import (
"strings"
"testing"

"github.com/dexpace/go-sdk/jsonl"
)

type rec struct {
N int `json:"n"`
}

func collectRecs(t *testing.T, input string) ([]rec, error) {
t.Helper()
var got []rec
var gotErr error
for v, err := range jsonl.Decode[rec](strings.NewReader(input)) {
if err != nil {
gotErr = err
break
}
got = append(got, v)
}
return got, gotErr
}

func TestDecodeNDJSON(t *testing.T) {
t.Parallel()

got, err := collectRecs(t, "{\"n\":1}\n{\"n\":2}\n{\"n\":3}\n")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(got) != 3 || got[0].N != 1 || got[1].N != 2 || got[2].N != 3 {
t.Fatalf("got %v, want n=1,2,3", got)
}
}

func TestDecodeSingleNoTrailingNewline(t *testing.T) {
t.Parallel()

got, err := collectRecs(t, "{\"n\":5}")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(got) != 1 || got[0].N != 5 {
t.Fatalf("got %v, want one value n=5", got)
}
}

func TestDecodeScalars(t *testing.T) {
t.Parallel()

var got []int
for v, err := range jsonl.Decode[int](strings.NewReader("1 2 3")) {
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
got = append(got, v)
}
if len(got) != 3 || got[0] != 1 || got[1] != 2 || got[2] != 3 {
t.Fatalf("got %v, want [1 2 3]", got)
}
}

func TestDecodeEmpty(t *testing.T) {
t.Parallel()

got, err := collectRecs(t, "")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(got) != 0 {
t.Fatalf("got %v, want no values", got)
}
}

func TestDecodeMalformedMidStream(t *testing.T) {
t.Parallel()

got, err := collectRecs(t, "{\"n\":1}\n{bad}\n")
if err == nil {
t.Fatal("expected a decode error for the malformed value")
}
if len(got) != 1 || got[0].N != 1 {
t.Fatalf("got %v, want the first value before the error", got)
}
}

func TestDecodeTruncated(t *testing.T) {
t.Parallel()

got, err := collectRecs(t, "{\"n\":1}\n{\"n\":")
if err == nil {
t.Fatal("expected an error for the truncated final value")
}
if len(got) != 1 || got[0].N != 1 {
t.Fatalf("got %v, want the first value before the truncation", got)
}
}

func TestDecodeEarlyBreak(t *testing.T) {
t.Parallel()

count := 0
for range jsonl.Decode[rec](strings.NewReader("{\"n\":1}\n{\"n\":2}\n{\"n\":3}\n")) {
count++
break
}
if count != 1 {
t.Fatalf("consumed %d values, want 1 after break", count)
}
}
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `go test ./jsonl/ -v`
Expected: FAIL — `jsonl.Decode` undefined.

- [ ] **Step 3: Create `jsonl/doc.go`**

```go
// jsonl/doc.go
// Copyright (c) 2026 dexpace and Omar Aljarrah.
// Licensed under the MIT License. See LICENSE in the repository root for details.

// Package jsonl decodes a stream of JSON values (JSON Lines / NDJSON) into typed
// values via a range-over-func iterator. Values may be separated by any JSON
// whitespace, so newline-delimited streams and concatenated values both work.
package jsonl
```

- [ ] **Step 4: Create `jsonl/jsonl.go`**

```go
// jsonl/jsonl.go
// Copyright (c) 2026 dexpace and Omar Aljarrah.
// Licensed under the MIT License. See LICENSE in the repository root for details.

package jsonl

import (
"encoding/json"
"errors"
"io"
"iter"
)

// Decode reads a stream of JSON values from r and yields each decoded into a T.
// Values may be separated by any JSON whitespace (newlines for NDJSON / JSON
// Lines, or none). Iteration ends at end of stream; a decode error is delivered
// as the second iteration value, after which iteration stops. The iterator is
// single-pass.
func Decode[T any](r io.Reader) iter.Seq2[T, error] {
return func(yield func(T, error) bool) {
dec := json.NewDecoder(r)
for {
var v T
err := dec.Decode(&v)
if errors.Is(err, io.EOF) {
return
}
if err != nil {
var zero T
yield(zero, err)
return
}
if !yield(v, nil) {
return
}
}
}
}
```

- [ ] **Step 5: Run tests to verify they pass**

Run: `go test ./jsonl/ -v`
Expected: PASS — NDJSON, single-value, scalars, empty, malformed, truncated, early-break.

- [ ] **Step 6: Commit**

```bash
git add jsonl/
git commit -m "feat(jsonl): add NDJSON/JSON-Lines stream decoder"
```

---

## Task 2: docs and full gate

**Files:**
- Modify: `doc.go`, `README.md`, `CLAUDE.md`

- [ ] **Step 1: Mention jsonl in `doc.go`**

Read `doc.go`. Within the `package dexpace` doc comment (single contiguous `//`
block; no second package clause / no duplicate header), add:

```go
// The jsonl package decodes streaming JSON Lines (NDJSON) responses into typed
// values.
```

- [ ] **Step 2: Add `jsonl` to `README.md`**

Read `README.md`. Add a `jsonl` row to the architecture/package table (matching
the column/link style): "JSON Lines / NDJSON streaming decoder (iter.Seq2)."

- [ ] **Step 3: Add `jsonl/` to `CLAUDE.md` Repository Layout**

Read `CLAUDE.md`. Add a `jsonl/` line near the other value-layer/streaming
packages: `jsonl/ # NDJSON / JSON Lines stream decoder`.

- [ ] **Step 4: Run the full gate**

Run:
```bash
gofmt -l .
go vet ./...
go test -race ./...
```
Expected: `gofmt -l .` prints nothing; `go vet` clean; every package passes under
the race detector (`jsonl` now has tests).

- [ ] **Step 5: Commit**

```bash
git add doc.go README.md CLAUDE.md
git commit -m "docs: document the jsonl package"
```

---

## Self-Review notes (for the implementer)

- **Spec coverage:** `Decode[T]` with EOF-clean / error-surface / break semantics
(Task 1); docs (Task 2).
- **Type consistency:** `jsonl.Decode[T](io.Reader) iter.Seq2[T, error]` used
identically across tasks/tests.
- **Clean EOF vs error:** `errors.Is(err, io.EOF)` ends cleanly;
`io.ErrUnexpectedEOF` (truncated value) and JSON syntax errors fall to the error
branch — verified by `TestDecodeTruncated` / `TestDecodeMalformedMidStream`.
- **Bounded:** `json.Decoder` reads incrementally; no whole-stream buffering added.
- **`make check`** green before opening the PR.
Loading
Loading