Java 25 Parallelism

<html>
<body>
<html><head></head><body><h1>Java 25: Actions serialize due to ForkJoinPool.commonPool parallelism dropping to 1 in containers</h1>
<h2>Summary</h2>
<p>When running Embabel on Java 25 inside containers (Docker/Kubernetes), all agent actions execute serially instead of in parallel. This is caused by Java 25's improved (more accurate) cgroup CPU detection, which reports <code>availableProcessors() = 1</code> in CPU-constrained containers. Since <code>ForkJoinPool.commonPool</code> sizes its parallelism based on this value, the effective parallelism drops to 1, serializing all concurrent agent operations.</p>
<p>This is not a bug in Embabel per se — Embabel correctly respects the JVM's reported processor count. However, it represents a significant behavioral regression for users upgrading to Java 25, and warrants documentation and potentially a defensive minimum parallelism floor.</p>
<h2>Reported Symptoms</h2>
<ul>
<li>Two concurrent requests each take ~2x as long to return (serialized execution)</li>
<li>Agents with multiple actions have each action executed sequentially</li>
<li>Problem only manifests on Java 25; previous JVM versions work as expected</li>
<li>Workaround: <code>-Djava.util.concurrent.ForkJoinPool.common.parallelism=4</code> restores parallel execution</li>
</ul>
<p>Reported by Jacqui Collier (Discord, 2/26/2026).</p>
<h2>Root Cause</h2>
<h3>Java 25 CPU detection change</h3>
<p>Prior to Java 25, the JVM was imprecise when reading container CPU limits — it often saw the host's full CPU count rather than the container's cgroup allocation. Java 25 corrects this to accurately respect cgroup CPU limits (e.g., Kubernetes <code>resources.limits.cpu</code>).</p>
<p>Related OpenJDK ticket: <a href="https://bugs.openjdk.org/browse/JDK-8362881">JDK-8362881</a></p>
<h3>Impact on ForkJoinPool</h3>
<p><code>ForkJoinPool.commonPool()</code> sizes itself as <code>availableProcessors() - 1</code>. When the container has no explicit CPU limit or is allocated 1 CPU:</p>

JVM Version | availableProcessors() | Common pool parallelism | Behavior
-- | -- | -- | --
Pre-25 | Host CPU count (e.g., 8) | 7 | Actions run in parallel
25 | Container CPU limit (e.g., 1) | 1 (minimum) | Actions serialize


<h3>Embabel's dependency on the common pool</h3>
<p><strong>Action item:</strong> Investigate and document exactly which Embabel components rely on <code>ForkJoinPool.commonPool</code> vs. custom executors. Candidates to audit:</p>
<ul>
<li><code>ExecutorAsyncer</code> / <code>Asyncer</code> — async action dispatch</li>
<li><code>ParallelToolLoop</code> — parallel tool execution</li>
<li><code>ScatterGather</code> / <code>aggregate</code> — fan-out/fan-in patterns</li>
<li><code>OperationScheduler</code> / <code>ProcessOptionsOperationScheduler</code> — action scheduling</li>
<li>Any use of Kotlin coroutines <code>Dispatchers.Default</code> (backed by <code>ForkJoinPool</code>)</li>
</ul>
<p>If Embabel delegates entirely to the common pool, this is purely a JVM/container configuration issue. If Embabel configures its own thread pools, those may also need to handle the <code>availableProcessors() = 1</code> case.</p>
<h2>Workarounds</h2>
<p>Users can apply any of the following immediately:</p>
<h3>Option 1: Set container CPU limits (preferred)</h3>
<p>Configure appropriate CPU limits so Java 25 detects them correctly:</p>
<pre><code class="language-yaml"># Kubernetes
resources:
  limits:
    cpu: "4"
  requests:
    cpu: "2"
</code></pre>
<pre><code class="language-bash"># Docker
docker run --cpus=4 ...
</code></pre>
<h3>Option 2: Override ForkJoinPool parallelism</h3>
<pre><code class="language-bash">java -Djava.util.concurrent.ForkJoinPool.common.parallelism=4 -jar agent.jar
</code></pre>
<h3>Option 3: Override JVM processor count detection</h3>
<pre><code class="language-bash">java -XX:ActiveProcessorCount=4 -jar agent.jar
</code></pre>
<blockquote>
<p><strong>Note:</strong> Option 1 is preferred because it provides accurate information to the JVM. Options 2 and 3 are overrides that may mask real resource constraints.</p>
</blockquote>
<h2>Proposed Actions</h2>
<h3>1. Documentation (high priority)</h3>
<p>Add a section to the deployment/operations guide covering:</p>
<ul>
<li>Java 25 container CPU detection behavior change</li>
<li>Recommended Kubernetes/Docker CPU configuration</li>
<li>JVM flag workarounds</li>
<li>How Embabel's parallel execution depends on <code>availableProcessors()</code></li>
</ul>
<h3>2. Audit parallelism dependencies (medium priority)</h3>
<p>Determine which Embabel components use <code>ForkJoinPool.commonPool</code> vs. custom executors, and document the dependency chain from <code>availableProcessors()</code> through to action execution.</p>
<h3>3. Consider a defensive parallelism floor (discussion)</h3>
<p>Evaluate whether Embabel should ensure a minimum parallelism level (e.g., 2-4) for its concurrent operations, rather than silently degrading to single-threaded execution. This could be:</p>
<ul>
<li>A configuration property: <code>embabel.agent.parallelism.minimum=2</code></li>
<li>A startup warning when <code>availableProcessors() == 1</code> and parallel features are in use</li>
<li>An auto-tuning default that uses <code>max(availableProcessors(), 2)</code></li>
</ul>
<p>Trade-off: a minimum floor improves resilience against container misconfigurations but could cause resource contention if the container genuinely only has 1 CPU.</p>
<h2>Environment</h2>
<ul>
<li><strong>Java version:</strong> 25 (EA or GA)</li>
<li><strong>Runtime:</strong> Containerized (Docker / Kubernetes)</li>
<li><strong>Container CPU config:</strong> Default or <code>limits.cpu: 1</code></li>
<li><strong>Embabel version:</strong> 0.3.x+</li>
</ul>
<h2>References</h2>
<ul>
<li><a href="https://bugs.openjdk.org/browse/JDK-8362881">JDK-8362881</a> — OpenJDK threading changes</li>
<li><a href="https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/util/concurrent/ForkJoinPool.html">ForkJoinPool javadoc</a> — common pool parallelism defaults</li>
<li>Discord report: Jacqui Collier, 2/26/2026</li>
<li>Community input: vidya333 (Discord, 2/26/2026) — identified root cause as JVM CPU detection</li>
</ul></body></html>
</body>
</html># Java 25: Actions serialize due to ForkJoinPool.commonPool parallelism dropping to 1 in containers

## Summary

When running Embabel on Java 25 inside containers (Docker/Kubernetes), all agent actions execute serially instead of in parallel. This is caused by Java 25's improved (more accurate) cgroup CPU detection, which reports `availableProcessors() = 1` in CPU-constrained containers. Since `ForkJoinPool.commonPool` sizes its parallelism based on this value, the effective parallelism drops to 1, serializing all concurrent agent operations.

This is not a bug in Embabel per se — Embabel correctly respects the JVM's reported processor count. However, it represents a significant behavioral regression for users upgrading to Java 25, and warrants documentation and potentially a defensive minimum parallelism floor.

## Reported Symptoms

- Two concurrent requests each take ~2x as long to return (serialized execution)
- Agents with multiple actions have each action executed sequentially
- Problem only manifests on Java 25; previous JVM versions work as expected
- Workaround: `-Djava.util.concurrent.ForkJoinPool.common.parallelism=4` restores parallel execution

Reported by Jacqui Collier (Discord, 2/26/2026).

## Root Cause

### Java 25 CPU detection change

Prior to Java 25, the JVM was imprecise when reading container CPU limits — it often saw the host's full CPU count rather than the container's cgroup allocation. Java 25 corrects this to accurately respect cgroup CPU limits (e.g., Kubernetes `resources.limits.cpu`).

Related OpenJDK ticket: [[JDK-8362881](https://bugs.openjdk.org/browse/JDK-8362881)](https://bugs.openjdk.org/browse/JDK-8362881)

### Impact on ForkJoinPool

`ForkJoinPool.commonPool()` sizes itself as `availableProcessors() - 1`. When the container has no explicit CPU limit or is allocated 1 CPU:

| JVM Version | `availableProcessors()` | Common pool parallelism | Behavior |
|---|---|---|---|
| Pre-25 | Host CPU count (e.g., 8) | 7 | Actions run in parallel |
| 25 | Container CPU limit (e.g., 1) | 1 (minimum) | Actions serialize |

### Embabel's dependency on the common pool

**Action item:** Investigate and document exactly which Embabel components rely on `ForkJoinPool.commonPool` vs. custom executors. Candidates to audit:

- `ExecutorAsyncer` / `Asyncer` — async action dispatch
- `ParallelToolLoop` — parallel tool execution
- `ScatterGather` / `aggregate` — fan-out/fan-in patterns
- `OperationScheduler` / `ProcessOptionsOperationScheduler` — action scheduling
- Any use of Kotlin coroutines `Dispatchers.Default` (backed by `ForkJoinPool`)

If Embabel delegates entirely to the common pool, this is purely a JVM/container configuration issue. If Embabel configures its own thread pools, those may also need to handle the `availableProcessors() = 1` case.

## Workarounds

Users can apply any of the following immediately:

### Option 1: Set container CPU limits (preferred)

Configure appropriate CPU limits so Java 25 detects them correctly:

```yaml
# Kubernetes
resources:
  limits:
    cpu: "4"
  requests:
    cpu: "2"
```

```bash
# Docker
docker run --cpus=4 ...
```

### Option 2: Override ForkJoinPool parallelism

```bash
java -Djava.util.concurrent.ForkJoinPool.common.parallelism=4 -jar agent.jar
```

### Option 3: Override JVM processor count detection

```bash
java -XX:ActiveProcessorCount=4 -jar agent.jar
```

> **Note:** Option 1 is preferred because it provides accurate information to the JVM. Options 2 and 3 are overrides that may mask real resource constraints.

## Proposed Actions

### 1. Documentation (high priority)

Add a section to the deployment/operations guide covering:

- Java 25 container CPU detection behavior change
- Recommended Kubernetes/Docker CPU configuration
- JVM flag workarounds
- How Embabel's parallel execution depends on `availableProcessors()`

### 2. Audit parallelism dependencies (medium priority)

Determine which Embabel components use `ForkJoinPool.commonPool` vs. custom executors, and document the dependency chain from `availableProcessors()` through to action execution.

### 3. Consider a defensive parallelism floor (discussion)

Evaluate whether Embabel should ensure a minimum parallelism level (e.g., 2-4) for its concurrent operations, rather than silently degrading to single-threaded execution. This could be:

- A configuration property: `embabel.agent.parallelism.minimum=2`
- A startup warning when `availableProcessors() == 1` and parallel features are in use
- An auto-tuning default that uses `max(availableProcessors(), 2)`

Trade-off: a minimum floor improves resilience against container misconfigurations but could cause resource contention if the container genuinely only has 1 CPU.

## Environment

- **Java version:** 25 (EA or GA)
- **Runtime:** Containerized (Docker / Kubernetes)
- **Container CPU config:** Default or `limits.cpu: 1`
- **Embabel version:** 0.3.x+

## References

- [[JDK-8362881](https://bugs.openjdk.org/browse/JDK-8362881)](https://bugs.openjdk.org/browse/JDK-8362881) — OpenJDK threading changes
- [[ForkJoinPool javadoc](https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/util/concurrent/ForkJoinPool.html)](https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/util/concurrent/ForkJoinPool.html) — common pool parallelism defaults
- Discord report: Jacqui Collier, 2/26/2026
- Community input: vidya333 (Discord, 2/26/2026) — identified root cause as JVM CPU detection

JVM Version	availableProcessors()	Common pool parallelism	Behavior
Pre-25	Host CPU count (e.g., 8)	7	Actions run in parallel
25	Container CPU limit (e.g., 1)	1 (minimum)	Actions serialize

Java 25 Parallelism #1454

Description

Java 25: Actions serialize due to ForkJoinPool.commonPool parallelism dropping to 1 in containers

Summary

Reported Symptoms

Root Cause

Java 25 CPU detection change

Impact on ForkJoinPool

Embabel's dependency on the common pool

Workarounds

Option 1: Set container CPU limits (preferred)

Option 2: Override ForkJoinPool parallelism

Option 3: Override JVM processor count detection

Proposed Actions

1. Documentation (high priority)

2. Audit parallelism dependencies (medium priority)

3. Consider a defensive parallelism floor (discussion)

Environment

References

Summary

Reported Symptoms

Root Cause

Java 25 CPU detection change

Impact on ForkJoinPool

Embabel's dependency on the common pool

Workarounds

Option 1: Set container CPU limits (preferred)

Option 2: Override ForkJoinPool parallelism

Option 3: Override JVM processor count detection

Proposed Actions

1. Documentation (high priority)

2. Audit parallelism dependencies (medium priority)

3. Consider a defensive parallelism floor (discussion)

Environment

References

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions