Service Unavailable: Portal and Endpoints

Incident Report for Pricemoov

Resolved

Issue fixed. DB is running properly
Posted Jul 10, 2025 - 19:34 EDT

Monitoring

Due to the unavailability of the initial database size during recovery, we launched another instance in degraded mode.

✅ APIs are operational
⚠️ Pricemoov Portal and Pricing Automation are still running with degraded performance

Our engineering team continues to monitor the situation closely and will provide further updates as we restore full service levels.

We appreciate your patience and understanding.
Posted Jul 10, 2025 - 11:43 EDT

Update

We attempted a manual reboot of our production database following stability issues. Unfortunately, the reboot operation did not complete successfully.

Our engineering team has identified that this issue now requires intervention from AWS.

We are working closely with our cloud partner and AWS to trigger a forced reboot and restore service as quickly as possible.

We will provide further updates as soon as we have confirmation from AWS.

Thank you for your continued patience.
Posted Jul 10, 2025 - 11:18 EDT

Update

Our engineering team has identified that the writer database instance remained in a “pending reboot” state following the upgrade, preventing full recovery and resumption of normal operations. This was due to a parameter group change requiring a manual reboot, as confirmed by our cloud provider's support team (DoiT/AWS).

We are now proceeding with a controlled reboot of the affected instance to complete the upgrade process and restore full functionality. Please note that no recent data changes have occurred during this maintenance window, and no unconfirmed transactions are at risk.
We expect the service to gradually recover shortly after the reboot. We will continue to monitor closely and provide another update as soon as the platform is fully operational again.

Thank you for your continued patience.
Posted Jul 10, 2025 - 10:55 EDT

Update

Our primary database is undergoing a planned upgrade. Both writer and reader instances are currently in the "Upgrading" state. We’ve confirmed there are no failure or rollback flags, indicating the upgrade process is still progressing normally.

Aurora upgrades can take extended time due to:

A distributed snapshot followed by WAL (write-ahead log) replay before upgrading system catalogs.

Sequential restarts of multiple instances, which increases total duration.

We are closely monitoring the upgrade and will provide updates as it completes. No data loss or failure has occurred. Thank you for your continued patience.
Posted Jul 10, 2025 - 09:38 EDT

Update

As database recovery is taking longer than expected, the Pricemoov Engineering team has initiated a dual recovery process. Starting at 14:00 UTC+2, we are restoring service from a secondary database instance to accelerate full recovery and minimize disruption.

We are closely monitoring the situation and will provide further updates shortly.

Thank you for your continued patience and understanding.
Posted Jul 10, 2025 - 08:32 EDT

Update

Database is now in the WAL (Write-Ahead Log) recovery phase before bringing the instance back online.
Posted Jul 10, 2025 - 08:21 EDT

Identified

We are currently experiencing an issue with our primary database, which is undergoing an automatic reboot following an elevated load condition. This may result in temporary unavailability or degraded performance of our services.

Our engineering team is actively monitoring the situation and working to restore normal operations as quickly as possible.

We will provide an update within the next 15 minutes or as soon as new information is available.

Thank you for your patience and understanding.
Posted Jul 10, 2025 - 08:01 EDT

Investigating

We are currently investigating this issue.
Posted Jul 10, 2025 - 06:50 EDT
This incident affected: Pricemoov Portal, API, and Pricing Automation.