WundertreOS - pgBouncer issues on some older projects – Incident details

SMS experiencing degraded performance

pgBouncer issues on some older projects

Resolved
Degraded performance
Started about 1 month agoLasted 22 hours 34 minutes

Affected

WundertreOS Database

Partial outage from 7:49 PM to 4:34 PM, Operational from 4:34 PM to 6:24 PM

Storage Buckets

Partial outage from 7:49 PM to 4:34 PM, Operational from 4:34 PM to 6:24 PM

Updates
  • Resolved
    UTC
    Resolved

    The fix has remained stable across all regions, and the issue has been resolved.

  • Monitoring
    UTC
    Monitoring

    A fix has been rolled out across all regions, and results are positive.
    We will continue monitoring to ensure the service remains stable.
    Moving this incident to monitoring.

  • Update
    UTC
    Update

    An enhanced automated solution is currently being tested in staging.
    It is expected to be more effective and efficient than the approach used previously.
    Once testing has completed successfully, we will use this approach to roll out the fix in production.
    We will provide further updates as this work progresses.

  • Update
    UTC
    Update

    We are continuing to rollout the fix to further regions.

  • Identified
    UTC
    Identified

    This issue is fixed in eu-central-2 and ap-south-1. We are continuing to rollout the fix to all regions.

  • Update
    UTC
    Update

    Some users on Project versions 15.1 and older are experiencing a pgBouncer configuration issue causing connection issues for the Dedicated Pooler and Storage services. We are continuing our investigation of this issue.

  • Investigating
    UTC
    Investigating

    Some users on older Projects are experiencing a pgBouncer configuration issue causing connection issues for the Dedicated Pooler and Storage services. This issue should only affect users with Project versions 15.1 and older. We have determined the cause of this issue and we are rolling out a fix.