Description
At 23:53 on Monday 17th January one of the two Firewalls within the shared infrastructure environment of the National Systems, impacting all National e-verification systems using Solidsoft Reply as IT-provider, partially failed. The Load Balancer, which passes transactions between the two firewalls, continued to pass transactions to the failed firewall as the failure did not affect the Load Balancer communications connection. However, the failure did prevent the affected server from passing these transactions on to SMVS.
The problem was escalated to a high priority incident in the morning the next day as NMVOs and end-users reported on the problem. The failing firewall was restarted, and the processes were being successfully performed again.
Timeline
- At 23:53 on Monday 17th January one of the two Firewalls within the shared infrastructure environment of the National Systems partially failed.
- At 06:59 on Tuesday 18th January 2021 a ticket was logged by one of the NMVOs concerning access to both the IQE and PRD National System environments. There were intermittently successful attempts to access the NMVO Portal, in the majority of attempts an error was received.
- At 08:10 the incident was escalated to a high priority incident and Solidsoft Reply created a team to investigate the cause.
- At 08:23 a Swedish end-user called in to e-VIS reporting that they experienced problem when trying to verify and decommission packs.
- At 09:01 following investigations by Solidsoft Reply it was identified that one of the Firewalls was not processing requests and a process to restart the firewall was implemented.
- At 09:19 the firewall was restarted, and Solidsoft Reply Operations monitored the system and this indicated that processes were being successfully performed.
- At 09:50 the incident was marked as resolved and all of the impacted NMVOs was notified.
- At 10:29 e-VIS communicated to the Swedish end-users and their IT-suppliers about the incident being resolved.