Issue Summary
Some customers reported that they were not able to access our merchant portal service between 11:50am and 1:37pm on Monday November 12. There were also reports of issues accessing the tyro mobile application between 1:25pm and 1:37pm.
Tyro Incident number
199072
Description of Impact
At around 12:30pm on Monday 12 November 2018, Tyro was alerted to issues with access to the Merchant Portal for some customers. Further investigation confirmed that the backend service to allow end user login was unavailable which resulted in new login attempts to fail.
Technical Root Cause
Improvements to the backend services for Merchant Portal was scheduled to be deployed on that morning. We rolled out these changes to the Primary site by following our standard process. This roll out was unsuccessful which our monitoring tool failed to detect. We then proceeded with rolling out the change to the secondary site that resulted in the backend user login service being unavailable.
Remediation
Once the issue was identified to be related to the changes being rolled out, a fix was implemented to resolve the issue.
Prevention
We held a post incident review and identified the following actions items to be implemented:
Raise awareness and knowledge on the criticality of the services that are a dependency for Merchant Portal and improve the detection and alerting capabilities.
