[ad_1]
At Singapore’s land checkpoints, travellers experienced delays of up to 30 minutes at the onset of the incident, said Assoc Prof Faishal. But at Woodlands Checkpoint, there was later another hour of delay for cars as the car arrival zone had to be converted to clear motorbikes manually, he added.
About 55,000 travellers passed through the two land checkpoints during the disruption.
“SEVERE SYSTEM OVERLOAD”
The disruption was a result of a “severe system overload” after a pre-scheduled trial for an upgrade of the Multi-Modal Biometrics System (MMBS), said Assoc Prof Faishal. The trial was conducted at about 10.40am on Mar 31.
The MMBS facilitates automated immigration clearance using travellers’ biometrics.
As a result, automated clearance lanes at all the departure halls in Changi Airport and certain automated lanes at Woodlands and Tuas Checkpoints were affected. The rest of the automated lanes, including those at the sea checkpoints, remained operational.
“Prior to the incident, ICA had been upgrading its systems progressively and cautiously, with 10 out of 12 systems enhanced. Only the MMBS and one other system had not yet been upgraded,” said Assoc Prof Faishal.
“For the MMBS, the system upgrade would involve replicating the large biometric database to an upgraded system. This has to be done continuously over a few days, and the MMBS has to remain operational during this period.”
As per standard procedure for any system upgrade, tests in the User Acceptance Test environment were conducted beforehand, said Assoc Prof Faishal. These tests were successful and subsequent trials in the production environment were also stable.
“Next, we had planned to conduct further controlled trials during different times of the day, to ascertain that the system upgrade would not disrupt operations during the wee hours, off-peak hours, and peak hours, before proceeding with the actual upgrade,” he added.
The “wee hours” trial was conducted on Mar 15 from 1.30am to 3.30am, and was successful.
“The trial on 31 March 2023, when the incident happened, was the off-peak hours trial, to take place from 10am to 2pm. The vendors were on standby on-site, and the plan was to recover the system within 30 minutes if the trial did not go well,” he explained.
“The trial caused the storage systems to overload at about 10.40am, and the process was aborted immediately. However, the extent of the overload was much more severe than anticipated, and the vendors who were on-site had to work with their global support team to diagnose and reboot the servers.”
The recovery process ultimately took about four-and-a-half hours and the MMBS recovered around 3pm.
ICA immediately activated its Business Continuity Plan once the system went down, said Assoc Prof Faishal. This meant that off-duty officers were recalled to help man manual immigration counters and perform crowd control.
“Across all the checkpoints, the failover process kicked in, and all the manual counters, and certain automated lanes, switched to backup systems. Not all the automated lanes have this failover capability, as different models were procured over the years,” he added.
“Travellers were re-directed to manual counters for immigration clearance. ICA immediately stepped-up manning of the manual counters through a combination of measures such as recalling off-duty officers, deploying administrative staff and retaining the officers from the outgoing shift.”
In addition, Changi Airport Group (CAG) assisted ICA by deploying additional Changi Youth Ambassadors and office staff, along with CAG’s duty terminal managers and their Customs, Immigration and Quarantine Team.
[ad_2]
Source link