WundertreOS - Notice history

All metrics have been nominal and stable for several hours now, and our upstream provider has resolved their incident.

We've received confirmation from our networking provider that there is recovery across all regions. We are able to corroborate this with our own metrics collections. We will continue to monitor for a bit, but we believe this to be resolved at this time.

We apologize for the trouble, and we appreciate your patience.

We've seen a marked improvement in error rates across all regions in the last few minutes. We are still waiting on confirmation from our provider on status, but we believe things to be improved at the moment. We'll keep working on this until we have for sure reached a resolution.

We recognize that some of our users, especially in parts of North and South America, are continuing to experience DNS and 5XX errors when trying to connect to their projects. We have escalated this issue with our network provider and will work with them continuously until the problems are resolved.

Some users continue to experience errors. We are raising the issue with our provider.

Our upstream provider has reported their incident resolved. We will continue to monitor for any errors our customers may be experiencing for another 30 minutes.

The fix that our upstream provider implemented has taken effect. Service has significantly improved across all previously impacted regions. There are still residual elevated error rates in a handful of regions and we will continue to work with our provider to eliminate them.

Our upstream provider has rolled the fix out to most regions. While we are seeing improvement, we are still seeing increased error rates in Mexico and Brazil and are continuing to work with them until this is fully resolved.

Upstream provider mitigations are still under way, and we are seeing improvements incrementally as routing regions have the mitigations applied. We are still seeing particular impact in South America, but some users in North America may still be affected as well. We anticipate continued incremental improvements and will update as we get more information.

Our upstream networking provider has identified a fix and is currently implementing a fix. We are seeing preliminary improvements to error rates for users in North America, but are still following this issue closely with our provider.

Our upstream networking partner is continuing their investigation. We will continue to post regular updates as we have more information.

Our upstream networking provider has discovered an issue and has declared an incident on their side. They are currently working toward resolution. We are actively working with them and will provide updates on progress as more information is available.

We continue to have active lines of investigation going with our upstream network provider partners, and are still working toward a resolution.

This issue affects the network-level access to projects. The projects themselves and the data in them are safe and unaffected.

Users are seeing two separate symptoms of these networking issues:

1. DNS Lookup failures for supabase project URLs
2. 530 responses to HTTP requests to supabase project API Endpoints

We continue to work with partners and investigate.

We have further clarified the scope and impact of the issues currently affecting users. A subset of users across North and South America are experiencing DNS resolution failures and HTTP 530 errors.
These symptoms indicate a networking-related issue impacting the availability of Supabase projects. We are actively working with both internal teams and external network providers to isolate the root cause and determine the most effective path to resolution.
We will continue to provide updates as more information becomes available.

We are actively investigating this issue and working with relevant partner organizations. At this time, we do not believe this issue is specific to any particular network provider. A subset of users continue to be impacted in both South and North America.

We are currently investigating reports of users experiencing login issues when using Supabase Auth.

At this time, the impact appears to be limited to a subset of users, primarily in North and South America.
The exact cause and scope are still being determined.

Our team is actively working to identify the root of the issue and will provide updates as more information becomes available.

We appreciate your patience while we investigate.

Our engineers have monitored the fix and confirmed the issue with EU subusers unable to send emails has now been resolved. All services are now operating normally at this time.

Our engineers have implemented a fix and are monitoring system performance. We will provide another update in an hour or as soon as more information becomes available.

We are continuing investigating this issue. We will provide more updates in 2 hours, or as soon as they become available.

Error rates have returned to normal. This incident has been resolved.

We carried out a controlled redeployment of the affected services to restore stability.
The system is now operating normally, and we will continue to monitor.

The fix was implemented. All impacted projects and newly created projects in the region are now successfully connecting to storage services.

The root cause, involving task instances and batch sending events, has been identified. Our engineering team is working on a fix.

We are continuing to investigate elevated error rates affecting storage services in the ap-northeast-1 (Tokyo) region.
While we are still observing a higher-than-normal level of errors, initial findings indicate that the overall impact is lower than first expected. Many affected requests are succeeding upon retry.
Our team is actively working to determine the root cause and will provide further updates as more information becomes available.

We are continuing to investigate elevated error rates affecting storage services in the ap-northeast-1 (Tokyo) region.
Newly created projects may still experience issues connecting to storage, while existing projects remain unaffected at this time.
Our team is actively working to determine the root cause and assess the full impact.
We will provide further updates as more information becomes available.

All storage operations have returned to normal.

The upstream incident has been resolved and we have not observed any additional errors. We will continue to monitor our storage infrastructure for continued stable operations.

We are observing storage error rates returning to normal.

We have identified the cause of the storage issue in EU-Central-2. The upstream provider is working to resolve the issue.

Both regions are now working as usual.

Response times in both the ap-northeast-1 (Tokyo) and ap-northeast-2 (Seoul) regions have now stabilised.
We will continue to monitor the systems closely to ensure stability.

Response times in the ap-northeast-1 (Tokyo) region have improved following mitigation efforts.
We are currently rolling out the fix to the ap-northeast-2 (Seoul) region.
Further updates will be provided as the rollout continues.

We are continuing to investigate reports of Edge Functions experiencing high response times in Asia Pacific regions
Our team is still working to determine the underlying cause.
We understand the disruption this may be causing and are treating this with high priority.

We are continuing to investigate reports of Edge Functions experiencing high response times in Asia Pacific regions
Our team is still working to determine the underlying cause.
We understand the disruption this may be causing and are treating this with high priority.

The issue affecting Edge Functions has now been fully resolved, and users have reported Edge Functions operating as expected.
The root cause was a denial-of-service event that led to elevated error rates and 504 responses across impacted regions.
Mitigations have been implemented to restore stability.
We will be publishing a full post-mortem with additional details, once our internal review is complete.
Thank you for your patience while we worked to resolve this issue.

The fix has been implemented and early indicators suggest it has resolved the 504 errors affecting Edge Functions.
We are continuing to monitor system performance closely to ensure stability and confirm full recovery across all affected regions.
We will provide a final update once we are confident the issue is fully resolved.

We believe we have identified the underlying cause of the 504 errors affecting Edge Functions across multiple regions.
The team are currently implementing corrective measures to address the issue and restore service stability.

We are continuing to investigate reports of Edge Functions returning 504 errors across multiple regions.
Our team is still working to determine the underlying cause.
We understand the disruption this may be causing and are treating this with high priority.