Follow one slow request before resizing

Choose a request with a clear expected result

For an illustrative report endpoint, record the route pattern, approximate data size, authentication context and whether it creates background work. Use a test account and representative sanitized data. Do not copy tokens, customer identifiers or complete request bodies into an incident note.

Separate first-request startup cost from repeated steady-state behavior. Record the UTC window and release revision. Determine whether all routes are slow, only large reports are affected, or the delay follows one external dependency. These distinctions make the next measurement more useful than a broad collection of unrelated metrics.

Build a timeline without double-counting

OpenTelemetry traces connect related spans through propagated context across service boundaries. Matching timestamps alone is not enough. Nested synchronous spans and parallel calls can overlap. Asynchronous children may outlive their parent, so adding every span duration can double-count elapsed time. Follow the operations that determine when the request completes.

Technical reference: OpenTelemetry traces · OpenTelemetry span end behavior.

The largest interval suggests where to inspect, not the root cause. A database interval measured in the API can include waiting for a connection before a query reaches the database. Missing telemetry also does not prove that no work occurred; inspect sampling and instrumentation coverage.

Correlate application and host observations

Collect request timings, errors, database connection usage, queue age, CPU, memory and disk pressure for the same window. Note concurrent backups, migrations or releases. A single utilization screenshot can miss the burst that affected the request.

Use correlation IDs and timestamps in redacted logs with appropriate retention and access controls. Avoid raw secrets, customer payloads and unbounded metric labels. Instrument the boundary needed for the hypothesis and review overhead. A new tracing agent is itself a change worth observing.

Inspect the query without causing another incident

Identify query shape, input-size characteristics and connection or lock waits first. PostgreSQL EXPLAIN describes a plan; EXPLAIN ANALYZE actually executes the statement and adds instrumentation overhead. A write can change data, while an expensive read still creates load. Use an isolated representative database for initial investigation.

Planner cost is not elapsed milliseconds. A tiny test table may yield a different plan from the application's larger data set. Review statistics, indexes and distribution, and compare the same query shape after changing it. Never paste an unknown write into an analysis command on production merely to obtain timing.

Technical reference: PostgreSQL EXPLAIN.

Predict what one change should improve

If pool waiting dominates, inspect worker connections or requests that hold a connection unnecessarily. If query execution dominates, inspect the plan and requested rows. If an external API dominates, review timeouts, retries and whether the work must be synchronous. CPU saturation during serialization points to a different experiment.

Write a prediction: reducing export concurrency should reduce API pool waiting while exports take longer. This exposes the tradeoff. Repeat the same request mix, compare latency and errors, and check that the improvement did not simply move failure into an ever-growing queue.

Keep the result and its uncertainty

Record the original observation, tested change, comparison method and remaining uncertainty. If evidence contradicts the hypothesis, undo the isolated change where appropriate and investigate another boundary. Avoid accumulating permanent tuning changes with no explanation.

Resize when measurements identify a resource constraint additional allocation could address. Use the workload budget and capture a release regression in the incident brief. The outcome is a reasoned diagnosis, not a promised latency or a provider benchmark.