what affect does removeOnComplete have in embedded mode?

[arthurvanl]

Do I need to set that on true in order to properly use embedded mode?

[egeominotti]

No, you don't need to set removeOnComplete: true to use embedded mode properly. But understanding what it does helps you decide.

What removeOnComplete does

removeOnComplete: false (default)

When a job completes:

1. Job stored in completedJobs set (max 50,000 entries, FIFO eviction)
2. Full job data stored in completedJobsData map (max 50,000)
3. Return value stored in jobResults LRU cache (max 10,000)
4. Job marked as completed in SQLite
5. Result written to SQLite results table

You can still query the job afterward:

const job = await queue.getJob(jobId);        // works
const result = await queue.getJobResult(jobId); // works
const state = await queue.getJobState(jobId);   // returns 'completed'

Memory is bounded automatically — when collections hit their max, oldest entries are evicted. So even without removeOnComplete, memory won't grow unbounded.

removeOnComplete: true

When a job completes:

1. Job immediately deleted from jobIndex
2. Job deleted from SQLite (deleteJob)
3. Not stored in completedJobs, completedJobsData, or jobResults
4. The completed event still fires normally

You cannot query the job afterward:

const job = await queue.getJob(jobId);    // returns undefined
const result = await queue.getJobResult(jobId); // returns undefined

When to use each

Scenario	Recommendation
Fire-and-forget jobs (don't need results)	removeOnComplete: true
High throughput (100k+ jobs/hour)	removeOnComplete: true — saves memory + disk
Need job results for chains/flows	removeOnComplete: false (default)
Need audit trail / job history	removeOnComplete: false (default)
Using getParentResult() in FlowProducer	Must be false — otherwise the result is deleted before the child can read it

For your use case

Since you're using chains with FlowProducer and SandboxedWorker, I'd recommend keeping the default (false) — at least for jobs that are part of a chain, because child steps need getParentResult() to access previous results.

For standalone jobs that don't need results, you can set it per-job:

await queue.add('cleanup', data, { removeOnComplete: true });  // fire-and-forget
await queue.add('fetch-data', data, { removeOnComplete: false }); // keep result for chain

Or set it as a default for the entire queue:

const queue = new Queue('my-queue', {
  defaultJobOptions: { removeOnComplete: true },
});

TL;DR

Embedded mode works fine with the default (false). Memory is already bounded (50k completed jobs max, auto-eviction). Only set true if you want to aggressively save memory/disk and don't need to query completed jobs.