The LeadDyno app and affiliate websites are now fully restored. There's still a temporary gap in tracking data from Wednesday morning EST until the time of the outage on Thursday morning EST. Please know that restoring this missing data is our utmost priority, though it may take a little time. We promise to keep you informed with a more specific timeline as soon as we have it.
Posted Oct 27, 2023 - 12:37 PDT
Update
We're thrilled to announce that the LeadDyno app and all affiliate websites are now back in action, running smoothly once again. We appreciate your patience and understanding during this time. We will keep the status in "monitoring" for the next couple hours to keep an eye on things.
While things are getting back on track, there's still a temporary gap in tracking data from Wednesday morning EST until the time of the outage on Thursday morning EST. Please know that restoring this missing data is our utmost priority, though it may take a little time. We promise to keep you informed with a more specific timeline as soon as we have it.
The tracking data from integrations and app scripts, which includes visitor information, leads, purchases, and affiliate payouts, was briefly on pause since the Thursday morning outage. But here's the good news – it's now actively being updated! You can expect to see all the tracking data from the moment of the outage up to the present in the app very soon. If you're using the REST API for tracking your data, kindly make API requests after 2023-10-25 04:14:35 EST.
We want to sincerely apologize for any inconvenience caused by this service interruption.
If you have any questions or need assistance, please reach out to our support team at support@leaddyno.com. Please bear in mind that there might be some delays in our response times due to the expected increase in requests related to this outage. Your continued patience is greatly appreciated.
Posted Oct 27, 2023 - 08:14 PDT
Monitoring
We want to extend our heartfelt thanks for your patience and understanding during the recent outages we encountered with the LeadDyno app and affiliate websites. Your support means the world to us, and we've been putting in every effort to encourage AWS to swiftly resolve the issues and bring everything back to normal.
Our aim is to have the app and all affiliate websites running smoothly again by 11am EST today. To achieve this, we're working with a version of the app from Wednesday morning EST. During this process, there will be an interruption in tracking data from Wednesday morning EST until the time of the outage on Thursday morning EST. Please rest assured that its our top priority to recover this missing data, though it may take a bit of time. We'll provide you with a more precise timeframe as soon as we have it.
The tracking data from integrations and app scripts, including visitors, leads, purchases, and affiliate payouts, was temporarily on hold since the Thursday morning outage. As we get the app up and running these will be updated. You can expect to see all the tracking data from the moment of the outage up to the present in the app very soon.
We completely understand the frustration this may have caused, and we are committed to providing you with an overview of what went wrong and our plans to prevent similar issues in the future, all without delving too deeply into technical jargon.
What Caused the Issue:
The problem stemmed from our use of a Multi-AZ cluster on AWS, which offers different zone availability. In this particular case, the cache size during a sync wasn't adequate due to the structure, leading to a rare bug in MySQL when trying to restore the data.
How We'll Prevent It From Happening Again:
We're taking proactive steps to ensure a smoother experience for you in the future. This includes adding a read-replica to the database and increasing the cache size.
Once again, we appreciate your patience and support during this challenging time. Your satisfaction is critical to us, and we're working tirelessly to ensure you have the best possible experience with LeadDyno.
Warmest regards, The LeadDyno Team
Posted Oct 27, 2023 - 06:54 PDT
Update
AWS action still required to address the issue. We are in constant communication with AWS and once a timeline is provided for a fix, we will provide an update here.
Posted Oct 26, 2023 - 19:57 PDT
Update
Still awaiting AWS actions to address the issue. No timeframe provided by AWS. Once a timeline is provided for a fix, we will provide an update there. Please know that we are monitoring through the evening.
Posted Oct 26, 2023 - 14:36 PDT
Update
Still awaiting AWS actions to address the issue. No timeframe provided by AWS.
Posted Oct 26, 2023 - 12:10 PDT
Update
We continue to be in communication with AWS and we are currently awaiting actions from their end to address the issue.
Posted Oct 26, 2023 - 10:09 PDT
Update
We are continuing to work on a fix for this issue.
Posted Oct 26, 2023 - 07:07 PDT
Identified
We regret to inform you that we are currently experiencing a service outage, resulting in a 503 Service Temporarily Unavailable error for both the app and affiliate websites.
*Current Status* We are actively working with AWS to address the root cause. You can check live updates on our Status Page. You can also subscribe on the Status Page to get automated updates.
*Tracking Unaffected* Tracking data remains secure. Your affiliate tracking should continue without significant disruption.
Our team is dedicated to resolving this issue as quickly as possible. We apologize for any inconvenience this may cause and will keep you updated on our progress.
Posted Oct 26, 2023 - 02:48 PDT
This incident affected: REST API, Management Portal, and Affiliate Desktops.