Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ LOCKOUT_DURATION_MINUTES=30
# Maximum size allowed for a secret value in bytes (default: 524288 = 512 KB)
SECRET_VALUE_SIZE_LIMIT_BYTES=524288

# Tokenization batch limit
# Maximum number of items in a batch tokenization request (default: 100)
TOKENIZATION_BATCH_LIMIT=100


# CORS configuration
# ⚠️ SECURITY WARNING: CORS is disabled by default for server-to-server API
Expand Down
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,26 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.28.0] - 2026-03-23

### Added
- Added configurable batch limit for tokenize and detokenize operations via `BATCH_LIMIT_TOKENIZATION` environment variable.
- Added individual transit key retrieval API (`GET /v1/transit/keys/{name}`) (#115).
- Added atomic batch tokenize and detokenize endpoints (`POST /v1/tokenization/tokenize/batch`, `POST /v1/tokenization/detokenize/batch`) (#119).
- Added individual tokenization key retrieval API by name (`GET /v1/tokenization/keys/{name}`) (#116).
- Added audit log filtering by `client_id` for `GET /v1/audit/logs` (#118).
- Added configurable Metrics Server timeouts (`METRICS_SERVER_READ_TIMEOUT`, `METRICS_SERVER_WRITE_TIMEOUT`, `METRICS_SERVER_IDLE_TIMEOUT`) (#114).
- Added database connection max idle time configuration (`DB_CONN_MAX_IDLE_TIME`) (#113).
- Added client secret rotation with automatic token revocation.
- Added strict capability validation for policies (#111).

### Changed
- Updated Transit Engine to support key deletion by name instead of UUID (#120).
- Updated Tokenization Engine to support key deletion by name (#117).

### Fixed
- Fixed rate limiter goroutine lifecycle and resource leaks (#112).

## [0.27.0] - 2026-03-06

### Added
Expand Down Expand Up @@ -441,6 +461,7 @@ If you are using `sslmode=disable` (PostgreSQL) or `tls=false` (MySQL) in produc
- Security model documentation
- Architecture documentation

[0.28.0]: https://github.com/allisson/secrets/compare/v0.27.0...v0.28.0
[0.27.0]: https://github.com/allisson/secrets/compare/v0.26.0...v0.27.0
[0.26.0]: https://github.com/allisson/secrets/compare/v0.25.0...v0.26.0
[0.25.0]: https://github.com/allisson/secrets/compare/v0.24.0...v0.25.0
Expand Down
2 changes: 1 addition & 1 deletion cmd/app/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import (

// Build-time version information (injected via ldflags during build).
var (
version = "v0.27.0" // Semantic version with "v" prefix (e.g., "v0.12.0")
version = "v0.28.0" // Semantic version with "v" prefix (e.g., "v0.12.0")
buildDate = "unknown" // ISO 8601 build timestamp
commitSHA = "unknown" // Git commit SHA
)
Expand Down
5 changes: 5 additions & 0 deletions conductor/archive/batch_limit_config_20260321/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Track batch_limit_config_20260321 Context

- [Specification](./spec.md)
- [Implementation Plan](./plan.md)
- [Metadata](./metadata.json)
8 changes: 8 additions & 0 deletions conductor/archive/batch_limit_config_20260321/metadata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"track_id": "batch_limit_config_20260321",
"type": "chore",
"status": "new",
"created_at": "2026-03-21T10:00:00Z",
"updated_at": "2026-03-21T10:00:00Z",
"description": "The TokenizeBatchRequest.Validate method uses a hardcoded value for the batch limit; this value must come from the global configuration."
}
27 changes: 27 additions & 0 deletions conductor/archive/batch_limit_config_20260321/plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Implementation Plan: Configurable Batch Limit for TokenizeBatchRequest

## Phase 1: Configuration [checkpoint: e43e7cc]
- [x] Task: Add TokenizationBatchLimit to Config struct and DefaultTokenizationBatchLimit constant in internal/config/config.go.
- [x] Task: Update `config.Load()` to load `TOKENIZATION_BATCH_LIMIT` from environment variables in `internal/config/config.go`.
- [x] Task: Update `config.Validate()` to include validation for `TokenizationBatchLimit` in `internal/config/config.go`.
- [x] Task: Update `.env.example` to include `TOKENIZATION_BATCH_LIMIT=100`.
- [x] Task: Conductor - User Manual Verification 'Configuration' (Protocol in workflow.md) e43e7cc

## Phase 2: DTO Updates [checkpoint: 86d62c6]
- [x] Task: Update `TokenizeBatchRequest.Validate` and `DetokenizeBatchRequest.Validate` in `internal/tokenization/http/dto/request.go` to accept `limit int`.
- [x] Task: Update validation rules to use `validation.Length(1, limit).Error(fmt.Sprintf("batch size exceeds limit of %d", limit))`.
- [x] Task: Update all tests calling these `Validate()` methods in `internal/tokenization/http/dto/request_test.go`.
- [x] Task: Conductor - User Manual Verification 'DTO Updates' (Protocol in workflow.md) 86d62c6

## Phase 3: Handler and DI Updates [checkpoint: 86d62c6]
- [x] Task: Update `TokenizationHandler` struct in `internal/tokenization/http/tokenization_handler.go` to include `batchLimit int`.
- [x] Task: Update `NewTokenizationHandler` in `internal/tokenization/http/tokenization_handler.go` to accept `batchLimit int`.
- [x] Task: Update `TokenizeBatchHandler` and `DetokenizeBatchHandler` in `internal/tokenization/http/tokenization_handler.go` to call `Validate(h.batchLimit)`.
- [x] Task: Update `initTokenizationHandler` in `internal/app/di_tokenization.go` to pass `c.config.TokenizationBatchLimit` to `NewTokenizationHandler`.
- [x] Task: Update `TokenizationHandler` tests in `internal/tokenization/http/tokenization_handler_test.go` to pass the batch limit to `NewTokenizationHandler`.
- [x] Task: Conductor - User Manual Verification 'Handler and DI Updates' (Protocol in workflow.md) 86d62c6

## Phase 4: Documentation
- [x] Task: Update `docs/configuration.md` with `TOKENIZATION_BATCH_LIMIT`.
- [x] Task: Update `docs/engines/tokenization.md` with information about the batch limit.
- [x] Task: Conductor - User Manual Verification 'Documentation' (Protocol in workflow.md)
23 changes: 23 additions & 0 deletions conductor/archive/batch_limit_config_20260321/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Specification: Configurable Batch Limit for TokenizeBatchRequest

## Overview
The `TokenizeBatchRequest.Validate` method currently uses a hardcoded value of 100 for the batch limit. This track aims to make this limit configurable via the global configuration using the `TOKENIZATION_BATCH_LIMIT` key, defaulting to 100 if not specified.

## Functional Requirements
- **Configuration Integration**: Add `TOKENIZATION_BATCH_LIMIT` to the global configuration structure and support its initialization from environment variables.
- **Dynamic Validation**: Update the `TokenizeBatchRequest.Validate` method to use the configured batch limit instead of the hardcoded value.
- **Error Message Update**: Ensure the error message returned when the limit is exceeded is `batch size exceeds limit of %d`, where `%d` is the current limit.
- **Update Documentation**: Update relevant documentation (e.g., `docs/configuration.md`, `docs/engines/tokenization.md`) to reflect the new configuration option.
- **Update `.env.example`**: Add `TOKENIZATION_BATCH_LIMIT` with the default value of 100 to the `.env.example` file.

## Acceptance Criteria
- [ ] `TOKENIZATION_BATCH_LIMIT` is successfully added to the configuration and can be set via an environment variable.
- [ ] The `TokenizeBatchRequest.Validate` method correctly uses the value from the configuration.
- [ ] If `TOKENIZATION_BATCH_LIMIT` is not set, the system defaults to a limit of 100.
- [ ] When a batch exceeds the limit, the error message correctly includes the configured limit value.
- [ ] Documentation (`docs/configuration.md`, `docs/engines/tokenization.md`) is updated.
- [ ] `.env.example` is updated.

## Out of Scope
- Modifying the core tokenization or batch processing logic.
- Adding limits to other batch operations beyond `TokenizeBatchRequest`.
5 changes: 5 additions & 0 deletions conductor/archive/release_v0.28.0_20260323/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Track release_v0.28.0_20260323 Context

- [Specification](./spec.md)
- [Implementation Plan](./plan.md)
- [Metadata](./metadata.json)
8 changes: 8 additions & 0 deletions conductor/archive/release_v0.28.0_20260323/metadata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"track_id": "release_v0.28.0_20260323",
"type": "chore",
"status": "new",
"created_at": "2026-03-23T12:00:00Z",
"updated_at": "2026-03-23T12:00:00Z",
"description": "Prepare to the next version v0.28.0"
}
33 changes: 33 additions & 0 deletions conductor/archive/release_v0.28.0_20260323/plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Implementation Plan: v0.28.0 Release Preparation

## Phase 1: Version Update & Changelog [checkpoint: 1052e1b]
- [x] Task: Update `version` to `v0.28.0` in `cmd/app/main.go` 165ae80
- [x] Task: Add v0.28.0 entry to `CHANGELOG.md` with all key changes: 61fdeda
- **New Feature:** Configurable batch limit for tokenize and detokenize operations.
- **Transit Engine:** Key deletion by name instead of UUID (#120).
- **Transit Engine:** Individual key retrieval API (#115).
- **Tokenization Engine:** Atomic batch tokenize and detokenize endpoints (#119).
- **Tokenization Engine:** Delete tokenization keys by name (#117).
- **Tokenization Engine:** Individual key retrieval API by name (#116).
- **Audit Logs:** Implement audit log filtering by `client_id` (#118).
- **Configuration:** Make Metrics Server timeouts configurable (#114).
- **Database:** Expose DB connection max idle time configuration (#113).
- **Auth:** Client secret rotation with automatic token revocation.
- **Auth:** Fix rate limiter goroutine lifecycle and resource leaks (#112).
- **Auth:** Implement strict capability validation for policies (#111).
- [x] Task: Conductor - User Manual Verification 'Phase 1: Version Update & Changelog' (Protocol in workflow.md) 1052e1b

## Phase 2: Documentation & OpenAPI Sync [checkpoint: a01f409]
- [x] Task: Run `make docs-lint` and address any issues.
- [x] Task: Audit `docs/openapi.yaml` and update it with new endpoints: cce88e6
- `/api/v1/tokenization/tokenize/batch` (POST)
- `/api/v1/tokenization/detokenize/batch` (POST)
- `/api/v1/tokenization/keys/{name}` (GET)
- `/api/v1/transit/keys/{name}` (GET)
- Verify audit log filtering params for `/api/v1/audit/logs`.
- [x] Task: Conductor - User Manual Verification 'Phase 2: Documentation & OpenAPI Sync' (Protocol in workflow.md) a01f409

## Phase 3: Final Verification [checkpoint: ec975f9]
- [x] Task: Run full test suite using `make test-all`. 60b5e6f
- [x] Task: Perform a final sanity check of the CHANGELOG and CLI version output. 60b5e6f
- [x] Task: Conductor - User Manual Verification 'Phase 3: Final Verification' (Protocol in workflow.md) ec975f9
32 changes: 32 additions & 0 deletions conductor/archive/release_v0.28.0_20260323/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Track Specification: v0.28.0 Release Preparation

**Overview:**
Prepare for the release of version v0.28.0. This includes updating the version number, generating the changelog for the significant features and fixes since v0.27.0, and performing final documentation/API audits.

**Functional Requirements:**
- **Update Version:** Update `version` in `cmd/app/main.go` and ensure all relevant locations reflect `v0.28.0`.
- **Update Changelog:** Generate and append release notes to `CHANGELOG.md` based on commits since `v0.27.0`.
- **Key Changes for v0.28.0:**
- **New Feature:** Configurable batch limit for tokenize and detokenize operations.
- **Transit Engine:** Key deletion by name instead of UUID (#120).
- **Transit Engine:** Individual key retrieval API (#115).
- **Tokenization Engine:** Atomic batch tokenize and detokenize endpoints (#119).
- **Tokenization Engine:** Delete tokenization keys by name (#117).
- **Tokenization Engine:** Individual key retrieval API by name (#116).
- **Audit Logs:** Implement audit log filtering by `client_id` (#118).
- **Configuration:** Make Metrics Server timeouts configurable (#114).
- **Database:** Expose DB connection max idle time configuration (#113).
- **Auth:** Client secret rotation with automatic token revocation.
- **Auth:** Fix rate limiter goroutine lifecycle and resource leaks (#112).
- **Auth:** Implement strict capability validation for policies (#111).
- **Documentation/OpenAPI Audit:** Run `make docs-lint` and ensure `docs/openapi.yaml` matches the current API state.

**Acceptance Criteria:**
- `cmd/app/main.go` reports version `v0.28.0`.
- `CHANGELOG.md` contains a comprehensive entry for v0.28.0 with all key changes correctly categorized.
- `make docs-lint` passes without errors.
- `docs/openapi.yaml` reflects the new batch tokenization, filtering, and key retrieval endpoints.

**Out of Scope:**
- Functional code changes (other than version bump).
- Deployment/Infrastructure changes.
2 changes: 2 additions & 0 deletions conductor/tech-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
## Data Persistence
- **PostgreSQL:** Primary relational database for production environments. Supported via `lib/pq`.
- **MySQL:** Alternative relational database support for broader infrastructure compatibility. Supported via `go-sql-driver/mysql`.
- **Connection Management:** Configurable connection pool settings including max open/idle connections, lifetime, and idle time for optimized resource usage.
- **Migrations:** [golang-migrate/migrate](https://github.com/golang-migrate/migrate) - Versioned database migrations for both PostgreSQL and MySQL.

## Cryptography & Security
Expand All @@ -16,6 +17,7 @@
- **Configurable Metrics Timeouts:** Environment-controlled Read, Write, and Idle timeouts for the Prometheus metrics server to prevent resource exhaustion.
- **Request Body Size Limiting:** Middleware to prevent DoS attacks from large payloads.
- **Rate Limiting:** Per-client and per-IP rate limiting middleware for DoS protection and API abuse prevention.
- **Tokenization Batch Limit:** Configurable limit for batch tokenization operations to ensure predictable performance and resource usage.
- **Secret Value Size Limiting:** Global limit on individual secret values to ensure predictable storage and memory usage.
- **Strict Capability Validation:** Centralized domain helpers for validating policy capabilities (`read`, `write`, `delete`, `encrypt`, `decrypt`, `rotate`) in CLI and API layers.
- **Secret Path Validation:** Strict naming rules for secret paths (alphanumeric, -, _, /) to ensure consistency and security.
Expand Down
3 changes: 3 additions & 0 deletions conductor/tracks.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Project Tracks

This file tracks all major tracks for the project. Each track has its own detailed plan in its respective folder.

---

9 changes: 8 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ LOCKOUT_DURATION_MINUTES=30

# Secret value size limit
SECRET_VALUE_SIZE_LIMIT_BYTES=524288

# Tokenization batch limit
TOKENIZATION_BATCH_LIMIT=100
```

## Database configuration
Expand Down Expand Up @@ -170,7 +173,11 @@ Logging level. Supported values: `debug`, `info`, `warn`, `error`, `fatal`, `pan

Maximum size allowed for a secret value in bytes (default: `524288` - 512 KB).

This limit is enforced before encryption and storage to prevent denial-of-service attacks using large secret payloads. If exceeded, the API returns a `413 Payload Too Large` error.
### TOKENIZATION_BATCH_LIMIT

Maximum number of items allowed in a single batch tokenization or detokenization request (default: `100`).

This limit is enforced to prevent resource exhaustion from excessively large batch operations. If exceeded, the API returns a `422 Unprocessable Entity` error.

## Master key configuration

Expand Down
4 changes: 2 additions & 2 deletions docs/engines/tokenization.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Example response (`201 Created`):
- **Endpoint**: `POST /v1/tokenization/keys/:name/tokenize-batch`
- **Capability**: `encrypt`
- **Body**: `items` (array of objects with `plaintext`, `metadata`, `ttl`).
- **Limit**: Maximum 100 items per batch.
- **Limit**: Maximum items per batch is configurable via `TOKENIZATION_BATCH_LIMIT` (default 100).

Generates tokens for multiple plaintext values in a single atomic operation. If any item fails (e.g., invalid format), the entire batch is rejected.

Expand Down Expand Up @@ -117,7 +117,7 @@ Example response (`200 OK`):
- **Endpoint**: `POST /v1/tokenization/detokenize-batch`
- **Capability**: `decrypt`
- **Body**: `{"tokens": ["string", "string"]}`
- **Limit**: Maximum 100 tokens per batch.
- **Limit**: Maximum tokens per batch is configurable via `TOKENIZATION_BATCH_LIMIT` (default 100).

Retrieves original plaintext values for multiple tokens in a single atomic operation.

Expand Down
38 changes: 38 additions & 0 deletions docs/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,44 @@ paths:
status:
type: string
example: healthy
/ready:
get:
tags: [auth]
summary: Readiness check
description: Comprehensive check that includes database connectivity.
responses:
"200":
description: Ready
content:
application/json:
schema:
type: object
properties:
status:
type: string
example: ready
components:
type: object
properties:
database:
type: string
example: ok
"503":
description: Not ready
content:
application/json:
schema:
type: object
properties:
status:
type: string
example: not_ready
components:
type: object
properties:
database:
type: string
example: error
/v1/token:
post:
tags: [auth]
Expand Down
6 changes: 5 additions & 1 deletion internal/app/di_tokenization.go
Original file line number Diff line number Diff line change
Expand Up @@ -334,5 +334,9 @@ func (c *Container) initTokenizationHandler(

logger := c.Logger()

return tokenizationHTTP.NewTokenizationHandler(tokenizationUseCase, logger), nil
return tokenizationHTTP.NewTokenizationHandler(
tokenizationUseCase,
c.config.TokenizationBatchLimit,
logger,
), nil
}
10 changes: 10 additions & 0 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ const (
DefaultLockoutDuration = 30 // minutes
DefaultMaxRequestBodySize = 1048576
DefaultSecretValueSizeLimit = 524288
DefaultTokenizationBatchLimit = 100
)

// Config holds all application configuration.
Expand Down Expand Up @@ -128,6 +129,8 @@ type Config struct {
MaxRequestBodySize int64
// SecretValueSizeLimitBytes is the maximum size of a secret value in bytes.
SecretValueSizeLimitBytes int
// TokenizationBatchLimit is the maximum number of items in a batch tokenization request.
TokenizationBatchLimit int
}

// Validate checks if the configuration is valid.
Expand Down Expand Up @@ -208,6 +211,7 @@ func (c *Config) Validate() error {
),
validation.Field(&c.MaxRequestBodySize, validation.Required, validation.Min(int64(1))),
validation.Field(&c.SecretValueSizeLimitBytes, validation.Required, validation.Min(1)),
validation.Field(&c.TokenizationBatchLimit, validation.Required, validation.Min(1)),
)
}

Expand Down Expand Up @@ -323,6 +327,12 @@ func Load() (*Config, error) {
"SECRET_VALUE_SIZE_LIMIT_BYTES",
DefaultSecretValueSizeLimit,
),

// Tokenization Batch Limit
TokenizationBatchLimit: env.GetInt(
"TOKENIZATION_BATCH_LIMIT",
DefaultTokenizationBatchLimit,
),
}

// Validate configuration
Expand Down
Loading
Loading