Skip to content
IncidentOS
Production
Incidents
SEV-2

Payment Service

Elevated Latency

payment-service

InvestigatingStarted

Scripted investigation is on the checkout-api SEV-1.

0.4%Errors
380msp95
8.2k/minreq/min
6%Impact

Metrics

Error rate

error_rate

Current error rate 0.4%. Peak 0.5%.

p95 latency

p95_latency

Current p95 383ms. Peak 413ms.

Request rate

request_rate

Current request rate 8.4k/min. Peak 8.4k/min.

Timeline

  1. Incident created

    Payment Service — Elevated Latency

Traces

showing 12 of 12

Trace IDTimeDurationStatus
14:29:10201msOK
14:07:32217msOK
13:58:13266msOK
13:47:04203msOK
13:12:36200msOK
12:52:51248msOK
12:47:47265msOK
12:45:29225msOK
12:43:18201msOK
12:19:09197msOK
12:11:10243msOK
12:00:30213msOK

Select a trace to open the waterfall.

Logs

showing 12 of 12

  • INFO

    POST /authorize ok duration=201ms

    payment-service · · 66083135a571

  • INFO

    POST /authorize ok duration=217ms

    payment-service · · 7e7cb17f7943

  • INFO

    POST /authorize ok duration=266ms

    payment-service · · 09fd36c8e8f8

  • INFO

    POST /authorize ok duration=203ms

    payment-service · · a49b155ebb10

  • INFO

    POST /authorize ok duration=200ms

    payment-service · · 89f6f1e926a2

  • INFO

    POST /authorize ok duration=248ms

    payment-service · · 2bd794faec19

  • INFO

    POST /authorize ok duration=265ms

    payment-service · · 980d44f88235

  • INFO

    POST /authorize ok duration=225ms

    payment-service · · 47bcf7df2c8f

  • INFO

    POST /authorize ok duration=201ms

    payment-service · · 0df90104b7af

  • INFO

    POST /authorize ok duration=197ms

    payment-service · · bbf60663cc13

  • INFO

    POST /authorize ok duration=243ms

    payment-service · · d023454c27e5

  • INFO

    POST /authorize ok duration=213ms

    payment-service · · f444cb86b783

Deployments

VersionTimeCommitChange
v1.18438268Issuer timeout budget 800ms
v1.1724355dIdempotency key on capture
v1.16cb4ce0Visa 3DS retry policy
v1.15866443Dead-letter queue for webhooks

Service graph

frontenderr 0.2% · p95 90ms
Healthy
checkout-apierr 18.4% · p95 2.8s
Critical
payment-serviceerr 0.4% · p95 380ms
Degraded
inventory-serviceerr 0.9% · p95 160ms
Degraded
user-serviceerr 0.1% · p95 70ms
Healthy
checkout-dbdependency of checkout-api
Healthy

frontend depends on checkout-api, which calls payment-service, inventory-service, and user-service. Database is shown as a checkout-api dependency.

Press Enter to send a message to the investigation agent.