WundertreOS - Notice history

This incident has been resolved.

We are continuing to monitor to ensure ongoing stability.

The remaining projects have now been brought back up, and we have no projects remaining on the unhealthy underlying hardware in the upstream infrastructure. We are monitoring to ensure ongoing stability.

Most projects have been brought back up, and the team is working on the remaining projects, which should be completed soon.

This issue was related to specific underlying hardware in the region, so this was isolated to only projects on that hardware. Project lifecycle operations (restarts, upgrades, etc) broadly across the region remain available.

All projects affected by the upstream infrastructure issues, and mitigation efforts are underway internally. These affected projects will be brought back up on new hardware as quickly as possible.

We have isolated the scope of the problem to a specific upstream incident. We are working to address the upstream incident and determine mitigations.

Project creation has returned to baseline consistently.

Project creation is reporting as recovered across the platform. We will monitor for stability in the near term.

We are seeing the mitigation steps having the desired effect. Project creation is recovering.

While the first set of actions are taking effect, we are implementing a second set of likely mitigations to attempt to facilitate improvement in project creation.

We continue to work this issue. The steps we have taken are addressing the hypothesized cause of the issue. We are implementing further related mitigations.

We have a working hypothesis on the cause of the issue, and are implementing steps to try to improve project creation performance.

The incident has been resolved and SMS delivery receipts from Twilio Long Codes to Liberty network subscribers in Puerto Rico are operating normally.

We have observed a recovery in SMS delivery receipts from Twilio Long Codes to Liberty network subscribers in Puerto Rico and are monitoring service stability. We will provide another update in 2 hours or as soon as more information becomes available.

Twilio customers may be experiencing SMS delivery receipt delays from Twilio Long Codes to Liberty network subscribers in Puerto Rico. Our team has identified the cause, and is working to resolve the issue. Message delivery may succeed, but delivery receipts may be delayed. We will provide another update in 24 hours or as soon as more information becomes available.

Twilio customers may be experiencing SMS delivery receipt delays from Twilio Long Codes to Liberty network subscribers in Puerto Rico. Our team has identified the cause, and is working to resolve the issue. Message delivery may succeed, but delivery receipts may be delayed. We will provide another update in 16 hours or as soon as more information becomes available.

Twilio customers may be experiencing SMS delivery receipt delays from Twilio Long Codes to Liberty network subscribers in Puerto Rico. Our team has identified the cause, and is working to resolve the issue. Message delivery may succeed, but delivery receipts may be delayed. We will provide another update in 8 hours or as soon as more information becomes available.

Twilio customers may be experiencing SMS delivery receipt delays from Twilio Long Codes to Liberty network subscribers in Puerto Rico. Our team has identified the cause, and is working to resolve the issue. Message delivery may succeed, but delivery receipts may be delayed. We will provide another update in 4 hours or as soon as more information becomes available.

Twilio customers may be experiencing SMS delivery receipt delays from Twilio Long Codes to Liberty network subscribers in Puerto Rico. Our team has identified the cause, and is working to resolve the issue. Message delivery may succeed, but delivery receipts may be delayed. We will provide another update in 2 hours or as soon as more information becomes available.

Twilio customers may be experiencing SMS delivery receipt delays from Twilio Long Codes to Liberty network subscribers in Puerto Rico. Our team has identified the cause, and is working to resolve the issue. Message delivery may succeed, but delivery receipts may be delayed. We will provide another update in 1 hour or as soon as more information becomes available.

The fix has remained stable across all regions, and the issue has been resolved.

A fix has been rolled out across all regions, and results are positive.
We will continue monitoring to ensure the service remains stable.
Moving this incident to monitoring.

An enhanced automated solution is currently being tested in staging.
It is expected to be more effective and efficient than the approach used previously.
Once testing has completed successfully, we will use this approach to roll out the fix in production.
We will provide further updates as this work progresses.

We are continuing to rollout the fix to further regions.

This issue is fixed in eu-central-2 and ap-south-1. We are continuing to rollout the fix to all regions.

Some users on Project versions 15.1 and older are experiencing a pgBouncer configuration issue causing connection issues for the Dedicated Pooler and Storage services. We are continuing our investigation of this issue.