Give background workers a bounded budget

Define one unit of useful work

For an internal CSV export, a request creates an export record; a worker reads a bounded data set, writes an output object and marks the record complete. Define maximum useful scope, time limit and cancellation behavior. Use an isolated queue, test database and destination that cannot notify real customers.

Separate computing time from database and storage waiting. One job-duration number hides these distinctions. Building the whole file in memory scales differently from streaming bounded batches. Keep the queue message small enough to describe the work without embedding customer data or a credential.

Make repeat delivery safe by design

Use a durable export identifier to relate attempts to one logical result. Enforce uniqueness and an atomic ownership/completion transition in durable state. Checking whether an export exists and then inserting in a separate unprotected step allows concurrent attempts to race. Publishing an object and acknowledging the queue message also form a failure boundary.

# Illustrative contract, not queue implementation
job_type: export-account-report
logical_result: EXPORT_RECORD_ID
input_scope: AUTHORIZED_ACCOUNT_AND_DATE_RANGE
attempt_limit: REVIEWED_FINITE_LIMIT
completion: ONE_PUBLISHED_RESULT_FOR_THIS_EXPORT
retry: CLASSIFIED_TRANSIENT_FAILURES_ONLY
failed_result: INSPECTABLE_WITHOUT_CUSTOMER_SECRETS

Celery connects late acknowledgement with idempotent tasks and documents cases where acknowledgement still happens after child-process termination. A queue option does not create exactly-once execution. Review your system's redelivery semantics and design the application result to tolerate retries.

Technical reference: Celery task behavior.

Set the budget before increasing processes

For a hypothetical worker using 300 MiB per active export, four simultaneous exports already imply about 1,200 MiB before runtime overhead. These are planning inputs, not benchmarks. Add database connections, query memory, temporary disk and output bandwidth; a process count is only one limit.

Rehearse with one active export and a realistic backlog. Increase to two while repeating the same API traffic. Compare completed useful exports, oldest-job age, latency, failures and host pressure. If throughput barely improves while database waits grow, stop increasing concurrency. Extra CPU may not remove that bottleneck.

Stop failure from creating more load

Classify errors before retrying. A temporary storage outage may be transient; an unauthorized account or unsupported export format needs a terminal error or intervention. Use a finite attempt budget and delayed retries with backoff and jitter where supported. Keep failed jobs inspectable with sensitive fields removed.

Apply timeouts to external calls and an overall task budget. Abandoning an attempt does not prove its remote side effect did not happen. A timed-out publish may already have written the output. Reconcile by export ID instead of publishing another result blindly.

Include workers in deployment and recovery

Stop new work on the old worker using its documented shutdown behavior. Let in-flight work finish or interrupt it safely under a known deadline. Test a crash after output is written but before completion is recorded; the replacement attempt should find a consistent result rather than duplicate it.

Keep message formats compatible across overlapping releases. A new API can enqueue a payload an old worker cannot read. Version the contract or sequence the rollout so supported consumers exist before new messages appear. Include these database writers in the schema compatibility review.

Choose the next constraint to change

Leave a concurrency setting, retry policy, task contract and measured stop rule. If expensive exports delay small jobs, consider separate queues with independent budgets before raising the global limit. If API latency suffers at any realistic export load, separating workers may be more useful than enlarging one shared host.

Repeat the same workload after a change and keep the comparison. The API and workers scenario explains where App 2 and extra memory enter the choice. These procedures do not imply a managed queue, unlimited jobs or automatic scaling.