Skip to content

perf: reduce memory usage of Command.output()#33335

Merged
bartlomieju merged 2 commits into
mainfrom
fix/command-output-memory-leak
Apr 21, 2026
Merged

perf: reduce memory usage of Command.output()#33335
bartlomieju merged 2 commits into
mainfrom
fix/command-output-memory-leak

Conversation

@bartlomieju
Copy link
Copy Markdown
Member

@bartlomieju bartlomieju commented Apr 20, 2026

Summary

Fixes RSS memory growth in Deno.Command.output() when called in a loop. Over 30k iterations, RSS grew from ~70MB to ~2GB while JS heap stayed flat at ~18MB.

The Command.output() path was creating ReadableStream wrappers (via readableStreamForRidUnrefable + readableStreamCollectIntoUint8Array) for stdout/stderr just to immediately collect all their data. The stream/controller machinery has significant overhead that causes RSS growth in tight loops.

The fix bypasses both ChildProcess and ReadableStream for the Command.output() hot path. spawnInner now:

  • Calls op_spawn_child directly instead of creating a ChildProcess instance
  • Reads stdout/stderr via core.read in a simple loop (readAllRid)
  • Avoids the overhead of stream creation, BYOB readers, and controller state

Before: 30k iterations -> RSS ~2GB
After: 30k iterations -> RSS ~215MB (stable)

Note: ChildProcess.output() (via .spawn().output()) is unchanged and still uses streams.

Test plan

Repro script from the issue:

for (let i = 1; i <= 30000; i++) {
  const command = new Deno.Command('echo', { args: ['hello'] });
  const result = await command.output();
  if (i % 5000 === 0) {
    const mem = Deno.memoryUsage();
    console.log(`${i}: RSS=${(mem.rss / 1024 / 1024).toFixed(1)}MB heap=${(mem.heapUsed / 1024 / 1024).toFixed(1)}MB`);
  }
}

Closes #33334
Closes #24674

🤖 Generated with Claude Code

The `Command.output()` path was routing stdout/stderr through
`readableStreamForRidUnrefable` + `readableStreamCollectIntoUint8Array`,
which leaked native memory from the ReadableStream/ByteStream controller
machinery. Over 30k iterations, RSS grew from ~70MB to ~2GB while heap
stayed flat at ~18MB.

The fix bypasses the ReadableStream layer entirely for `Command.output()`.
Instead of creating streams just to immediately collect all their data,
`spawnInner` now reads directly from the resource IDs using `core.read`
in a simple loop. This also avoids creating a ChildProcess instance,
eliminating the `#status` promise chain that captured `this` and
prevented GC of native resources.

Before: 30k iterations -> RSS ~2GB
After:  30k iterations -> RSS ~215MB (stable)

Closes #33334
Closes #24674

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bartlomieju bartlomieju force-pushed the fix/command-output-memory-leak branch from 109ab6a to 7e547e7 Compare April 20, 2026 19:36
@bartlomieju bartlomieju changed the title fix(process): memory leak in Command.output() fix(: memory leak in Command.output() Apr 20, 2026
@bartlomieju bartlomieju changed the title fix(: memory leak in Command.output() fix( memory leak in Command.output() Apr 20, 2026
@bartlomieju bartlomieju changed the title fix( memory leak in Command.output() fix: memory leak in Command.output() Apr 20, 2026
@bartlomieju bartlomieju requested a review from nathanwhit April 20, 2026 19:52
Copy link
Copy Markdown

@miracatbot miracatbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I took a close look at the new Command.output() fast path in ext/process/40_process.js.

The change makes sense to me:

  • it bypasses the ReadableStream wrapper machinery for the hot Command.output() case,
  • reads stdout/stderr directly with core.read() in a simple loop,
  • still waits for process completion in parallel,
  • and preserves the existing user-facing behavior around piped vs non-piped stdio.

I didn't spot a concrete correctness issue in the new resource-lifetime or output-collection logic, and the motivation / repro is compelling.

Copy link
Copy Markdown
Member

@nathanwhit nathanwhit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the PR description isn't quite right but the change is a perf improvement and seems right, so LGTM

Comment thread ext/process/40_process.js Outdated
@nathanwhit nathanwhit changed the title fix: memory leak in Command.output() perf: reduce memory usage of Command.output() Apr 20, 2026
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bartlomieju bartlomieju enabled auto-merge (squash) April 21, 2026 07:17
@bartlomieju bartlomieju merged commit 7f84f6c into main Apr 21, 2026
113 checks passed
@bartlomieju bartlomieju deleted the fix/command-output-memory-leak branch April 21, 2026 07:28
bartlomieju pushed a commit that referenced this pull request May 14, 2026
`Command.output()`'s fast path (`spawnInner` in
`ext/process/40_process.js`) bypasses `ChildProcess`. When that path was
introduced in #33335 it silently dropped the `signal` option during
destructuring, so passing an `AbortSignal` to `new Deno.Command()` and
then calling `.output()` no longer killed the child when the signal
fired.

Fixes #34066

Co-authored-by: lunadogbot <lunadogbot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory usage growth from command.output() Possible memory leak using Deno.Command

3 participants