WundertreOS - Elevated error rates affecting storage services in the ap-northeast-1 (Tokyo) region – Incident details

Storage Buckets experiencing partial outage

Elevated error rates affecting storage services in the ap-northeast-1 (Tokyo) region

Resolved
Degraded performance
Started 4 months agoLasted about 2 hours

Affected

WundertreOS Database

Operational from 4:27 PM to 4:27 PM, Degraded performance from 4:27 PM to 6:21 PM

Storage Buckets

Operational from 4:27 PM to 4:27 PM, Degraded performance from 4:27 PM to 6:21 PM

Updates
  • Resolved
    UTC
    Resolved

    The fix was implemented. All impacted projects and newly created projects in the region are now successfully connecting to storage services.

  • Identified
    UTC
    Identified

    The root cause, involving task instances and batch sending events, has been identified. Our engineering team is working on a fix.

  • Update
    UTC
    Update

    We are continuing to investigate elevated error rates affecting storage services in the ap-northeast-1 (Tokyo) region.
    While we are still observing a higher-than-normal level of errors, initial findings indicate that the overall impact is lower than first expected. Many affected requests are succeeding upon retry.
    Our team is actively working to determine the root cause and will provide further updates as more information becomes available.

  • Update
    UTC
    Update

    We are continuing to investigate elevated error rates affecting storage services in the ap-northeast-1 (Tokyo) region.
    Newly created projects may still experience issues connecting to storage, while existing projects remain unaffected at this time.
    Our team is actively working to determine the root cause and assess the full impact.
    We will provide further updates as more information becomes available.

  • Investigating
    UTC
    Investigating

    We are currently investigating elevated error rates affecting storage services in the ap-northeast-1 (Tokyo) region.
    Newly created projects may experience issues connecting to storage, while existing projects remain unaffected at this time.
    We are actively working to understand the full scope of the impact and will provide further updates as more information becomes available.