SEV-1

Checkout API

Elevated Error Rate

checkout-api

InvestigatingStarted 13:55

18.4%Errors

2.8sp95

24.1k/minreq/min

32%Impact

Metrics

Error rate

error_rate

Error rate rose from 0.8% to 18.4% after v2.31 at 13:45. Current 18.3%.

p95 latency

p95_latency

p95 rose from 420ms to 2.8s after the v2.31 deploy. Current 2.8s.

Request rate

request_rate

Traffic rose from 20k/min to 24.1k/min. The increase is too small to explain the error-rate jump. Current 24.1k/min.

Timeline

13:45
Deployment v2.31
checkout-api shipped Optimize checkout query
13:47
p95 latency begins increasing
Checkout p95 leaves the 420ms baseline
13:49
DB latency increases
db.query duration climbs on checkout-api
13:50
Error rate increases
Checkout error rate leaves the 0.8% baseline
13:52
HTTP 500 rate spikes
Timeouts surface as 500s on /checkout
13:53
Alert triggered
Error-rate alert fires for checkout-api
13:55
Incident created
SEV-1 Checkout API — Elevated Error Rate

Traces

showing 14 of 14

Time	Duration	Status
13:53:42	3.82s	ERROR
13:56:07	2.98s	ERROR
13:54:18	3.41s	ERROR
14:30:01	3.43s	ERROR
14:28:59	3.34s	ERROR
14:27:59	3.06s	ERROR
14:26:57	1.26s	ERROR
14:19:51	3.43s	ERROR
14:16:48	1.07s	ERROR
14:11:50	3.57s	ERROR
13:55:35	3.03s	ERROR
13:15:00	3.81s	ERROR
12:38:34	3.88s	ERROR
12:25:20	3.61s	ERROR

Select a trace to open the waterfall. Representative failed trace is .

Logs

showing 54 of 54

14:30:01.595ERROR
HTTP 500 POST /checkout duration=3428ms
checkout-api · · 284e1e77a0aa
14:30:01.592ERROR
database query exceeded 2s timeout
checkout-api · · 9f577ee2323d
14:30:01.590ERROR
context deadline exceeded
checkout-api · · 9f577ee2323d
14:30:01.587ERROR
request failed trace_id=2e5305ea6abe4d09 span_id=284e1e77a0aa
checkout-api · · 284e1e77a0aa
14:28:59.491ERROR
HTTP 500 POST /checkout duration=3336ms
checkout-api · · 8fe4eaa23a85
14:28:59.488ERROR
database query exceeded 2s timeout
checkout-api · · fa0dc89e517e
14:28:59.486ERROR
context deadline exceeded
checkout-api · · fa0dc89e517e
14:28:59.483ERROR
request failed trace_id=025cf66d8b200cd7 span_id=8fe4eaa23a85
checkout-api · · 8fe4eaa23a85
14:27:59.164ERROR
HTTP 500 POST /checkout duration=3064ms
checkout-api · · e619fad9a716
14:27:59.161ERROR
database query exceeded 2s timeout
checkout-api · · a0d02144afb3
14:27:59.159ERROR
context deadline exceeded
checkout-api · · a0d02144afb3
14:27:59.156ERROR
request failed trace_id=4773bfb802bfd891 span_id=e619fad9a716
checkout-api · · e619fad9a716
14:26:57.620ERROR
payment.authorize declined or timed out
checkout-api · · eb7c3681e629
14:26:57.616ERROR
request failed trace_id=1cad3a24d34ee541 span_id=eb7c3681e629
checkout-api · · eb7c3681e629
14:19:51.233ERROR
HTTP 500 POST /checkout duration=3433ms
checkout-api · · 74a29f91a790
14:19:51.230ERROR
database query exceeded 2s timeout
checkout-api · · 1f93621cd404
14:19:51.228ERROR
context deadline exceeded
checkout-api · · 1f93621cd404
14:19:51.225ERROR
request failed trace_id=908f68b86c0aa916 span_id=74a29f91a790
checkout-api · · 74a29f91a790
14:16:48.530ERROR
payment.authorize declined or timed out
checkout-api · · 4bd1abe928bd
14:16:48.526ERROR
request failed trace_id=e55cc8629c1da5d3 span_id=4bd1abe928bd
checkout-api · · 4bd1abe928bd
14:11:50.945ERROR
HTTP 500 POST /checkout duration=3574ms
checkout-api · · fa5e9785fc3d
14:11:50.942ERROR
database query exceeded 2s timeout
checkout-api · · 5d904316f770
14:11:50.940ERROR
context deadline exceeded
checkout-api · · 5d904316f770
14:11:50.937ERROR
request failed trace_id=5c7b0c2d5c604659 span_id=fa5e9785fc3d
checkout-api · · fa5e9785fc3d
13:56:07.126ERROR
HTTP 500 POST /checkout duration=2980ms
checkout-api · · 65fc9fc12ba1
13:56:07.123ERROR
database query exceeded 2s timeout
checkout-api · · b6ac113c77bc
13:56:07.121ERROR
context deadline exceeded
checkout-api · · b6ac113c77bc
13:56:07.118ERROR
request failed trace_id=bc73d11e90aa44c2 span_id=65fc9fc12ba1
checkout-api · · 65fc9fc12ba1
13:55:35.026ERROR
HTTP 500 POST /checkout duration=3031ms
checkout-api · · 1c8d1d08b5a5
13:55:35.023ERROR
database query exceeded 2s timeout
checkout-api · · 47a5827ebd71
13:55:35.021ERROR
context deadline exceeded
checkout-api · · 47a5827ebd71
13:55:35.018ERROR
request failed trace_id=106ad415a27388a0 span_id=1c8d1d08b5a5
checkout-api · · 1c8d1d08b5a5
13:54:18.448ERROR
HTTP 500 POST /checkout duration=3410ms
checkout-api · · dcbafda1aabb
13:54:18.445ERROR
database query exceeded 2s timeout
checkout-api · · 77614a57a2c4
13:54:18.443ERROR
context deadline exceeded
checkout-api · · 77614a57a2c4
13:54:18.440ERROR
request failed trace_id=9a31d42c18e07b55 span_id=dcbafda1aabb
checkout-api · · dcbafda1aabb
13:53:42.929ERROR
HTTP 500 POST /checkout duration=3820ms
checkout-api · · 8fd3c21a0001
13:53:42.926ERROR
database query exceeded 2s timeout
checkout-api · · 93ab4e21f006c8
13:53:42.924ERROR
context deadline exceeded
checkout-api · · 93ab4e21f006c8
13:53:42.921ERROR
request failed trace_id=8fd3c21a9b4d12ef span_id=8fd3c21a0001
checkout-api · · 8fd3c21a0001
13:53:00.000ERROR
Alert triggered: checkout-api error rate exceeded 5% threshold
checkout-api
13:52:00.000ERROR
HTTP 500 rate spike on POST /checkout
checkout-api
13:15:00.732ERROR
HTTP 500 POST /checkout duration=3811ms
checkout-api · · 5d3d93b37754
13:15:00.729ERROR
database query exceeded 2s timeout
checkout-api · · c53fa79f0c9a
13:15:00.727ERROR
context deadline exceeded
checkout-api · · c53fa79f0c9a
13:15:00.724ERROR
request failed trace_id=b8b28ef893c65244 span_id=5d3d93b37754
checkout-api · · 5d3d93b37754
12:38:34.429ERROR
HTTP 500 POST /checkout duration=3878ms
checkout-api · · e7fda381284d
12:38:34.426ERROR
database query exceeded 2s timeout
checkout-api · · 75e232181279
12:38:34.424ERROR
context deadline exceeded
checkout-api · · 75e232181279
12:38:34.421ERROR
request failed trace_id=f59acc3aec9b2de8 span_id=e7fda381284d
checkout-api · · e7fda381284d
12:25:20.513ERROR
HTTP 500 POST /checkout duration=3610ms
checkout-api · · 99f58166cf53
12:25:20.510ERROR
database query exceeded 2s timeout
checkout-api · · 655fece2aa12
12:25:20.508ERROR
context deadline exceeded
checkout-api · · 655fece2aa12
12:25:20.505ERROR
request failed trace_id=98e8645ef46a0ee0 span_id=99f58166cf53
checkout-api · · 99f58166cf53

Deployments

Version	Time	Commit	Change
v2.31correlated	13:45	a91f2c	Optimize checkout query
v2.30	09:21	83af31	Payment retry improvements
v2.29	yesterday 16:04	824820	Cart validation timeout padding
v2.28	yesterday 14:22	ba1432	Reduce checkout span cardinality
v2.27	3d ago 11:09	a5855c	Feature flag for express checkout
v2.26	4d ago 19:41	97e18c	Address tax calculation edge case
v2.25	7d ago 09:15	6b2384	Checkout session TTL increase

Service graph

frontenderr 0.2% · p95 90ms

Healthy

checkout-apierr 18.4% · p95 2.8s

Critical

payment-serviceerr 0.4% · p95 380ms

Degraded

inventory-serviceerr 0.9% · p95 160ms

Degraded

user-serviceerr 0.1% · p95 70ms

Healthy

checkout-dbdependency of checkout-api

Healthy

frontend depends on checkout-api, which calls payment-service, inventory-service, and user-service. Database is shown as a checkout-api dependency.