Issue Summary
On June 14th between 1.01am – 2.12am some customers experienced issues with reporting on the terminal and merchant portal.
Tyro incident number
283648
Description
During the affected time, a small number of our customers were unable to see reports in the Merchant Portal and on terminals. Between 1.47am and 6am, paper roll ordering service was impacted.
Technical root cause
One of Tyro’s datacenters lost connectivity to its provider during their scheduled maintenance. The provider’s failover mechanism did not activate properly and Tyro’s failover mechanisms did not account for this particular scenario. This led to an unintended outage for Tyro’s reporting services.
Communication to customers
Tyro’s status page was updated during the identification, monitoring and resolution stages of the incident.
Remediation
We shut down the reporting service in the affected datacentre and redirected traffic to the unaffected datacentre.
Prevention
Tyro maintains multiple levels of redundancy. Having a redundant provider link enables us to re-route traffic to minimise the impact to our customers. Pre-emptively switching over to a redundant provider link not affected by the provider’s scheduled maintenance, will help mitigate the impact.