Was it a crash or a hack?
The BC Ferries call centre, website and app were out of service for nearly nine hours during the morning and afternoon on Victoria Day. The ferry corporation found the root cause but censored it from reports and emails released due to a freedom of information request.
Interim chief information officer Steve Harris provided CEO Nicolas Jimenez with a three-page briefing note on the situation more than 30 hours later on May 23. But it was withheld in its entirety.
BC Ferries claimed it needed to censor documents for fear of harm to the security of a computer or communications system and harm to public safety. Some of the documents also contained policy advice or recommendations.
The heavily censored, detailed timeline showed the problem began at 5:26 a.m. on May 22. Someone left a voicemail for Jimenez at 7:53 a.m. to inform him of the tech trouble. Unix operating system technicians were called at 8:30 a.m. for support. At 10:55 a.m., data flowed to highway signs, and media message systems were returning online by 11:02 a.m. "Few hrs away from restored service," said the entry.
At 11:44 a.m., technicians estimated service would resume in three hours.
Throughout the morning, BC Ferries relied on Twitter to publish schedule updates. Public affairs executive director Deborah Marshall chimed in at 9:44 a.m., mentioning how she told reporters the issue was identified and technicians "rebooted the system."
Just over an hour later, Marshall asked colleagues for an update. "Media getting rather aggressive." She told a producer at Global BC at 11:13 a.m. that the ferries website was "slowly coming back online now."
The timeline said the website was changed to "current conditions lite" to display basic information. But, at 1 p.m., an unrelated issue. Departure Bay said it was ticketing for the 9:05 p.m. sailing. Just two minutes later, new bookings and redemptions were flowing.
At 1:45 p.m., a "current conditions is unavailable" message showed on the web and ops application. Almost an hour later, at 2:44 p.m., the mobile app and current conditions page were reported working, and emails were being received.
The website opened to the public at 2:52 p.m. Within minutes, 1,100 users were on the website and more than 600 on the app.
Below the timeline, the list of 12 observations was censored entirely.
Marshall told executives at 3:43 p.m. that reporters were asking what caused the outage. "Do we have an answer yet?"
Angela Soucie, the director of consumer marketing and digital experience, replied. "Maybe we keep it generic. It was nothing to do with the website, but a backend system failure that brought down several systems."
The next morning, 24 hours after it began, the executive director of BC Ferries's internal audit office asked Harris for a copy of whatever email he had prepared for executives about the root cause.
"Typically I am cc'd and/or included on any correspondence related to significant incidents. I received the service desk notices but they don't describe the root cause, only that the root cause was identified," wrote Christy Hermans,
Brian Anderson, the vice-president of strategy and community engagement, sent a script at noon on May 23 to Marshall and Jimenez titled "Key Messages - May Long Weekend." Such a document is standard for answering media questions.
In it, the computer system problem was called "unanticipated."
"The minute we experienced the problem, we began to address it. All available internal resources were made available. But it was complicated and despite many resources working on the issue, it took us nine hours to resolve."
The script said the company was "undertaking a deep dive to determine if the problem could have been avoided or dealt with earlier" and "undertaking broader work to revisit our technical environment to make sure it's resilient for our business needs. That's not a short-term fix."
The same briefing note said BC Ferries carried 400,000 passengers over the long weekend and was "deeply sorry" to those inconvenienced. Passengers were stranded on Bowen Island due to crew shortages and offered a water taxi to Horseshoe Bay. The script called the labour crunch the number one issue for BC Ferries, which had hired 800 people, but needed 100 more.
Staffing continued to wreak havoc throughout the Canada Day long weekend. There were more sailing waits than normal after the Coastal Celebration was taken out of Tsawwassen-to-Swartz Bay service for an "unplanned, extended refit" at Â鶹´«Ã½Ó³»Drydock in North Vancouver.
The website also continued to struggle. There was heavy traffic from the public seeking schedules and online booking.
According to the most recent annual report, for 2021-2022, BC Ferries spent $7.4 million on upgrading computer and communications equipment.
