CommsDay Story of the Week - Triple Zero continues

Our CommsDay story of the week has emerged over a long period and has reached an important phase with the release of the report of the Independent Review commissioned by Optus (the Schott Report) yesterday.  Today's coverage by CommsDay is thorough and detailed spread over three articles by Grahame Lynch.  The Optus Board has accepted all of the recommendations in the Schott Report.  Perhaps no surprise there.  Apart from the very serious consequences that can be associated with failures in the emergency services system, the Tripe Zero story shows once again the ways in which complex procedures involving decentralised decision-making can fail in high tech environments.

Sleeper issue in Optus outage: why so many Triple Zero calls failed to camp on

Beyond the operational failures that triggered Optus’ 18 September outage, Dr Kerry Schott’s independent review released yesterday identified a deeper and unresolved problem: why Triple Zero calls failed to reliably “camp on” to alternative networks or pathways once the Optus gateway was unavailable.

Of the 605 emergency call attempts during the outage, only 150 connected. 66 camped on to Telstra or TPG networks. 84 found another path within Optus. The majority never connected at all.
Under Australia’s emergency call framework, a handset unable to attach to its home network is supposed to automatically connect to any available network. Schott said understanding why this mechanism failed is now one of
the most urgent issues arising from the incident.

Ordinary voice calls briefly failed and then re-registered on secondary gateways. Triple Zero calls behaved differently. Each emergency call required a fresh registration attempt. When the primary gateway was unavailable, devices had to determine whether to try another gateway, another carrier or another radio technology. That decision depended on a complex interaction between device software, network timers and fallback availability.

During the outage, that interaction repeatedly broke down. A central factor identified by the review was Optus’ emergency inactivity timer, set at just 10 seconds at the time of the incident. The timer governs how long a de- vice actively attempts to connect before abandoning the attempt and trying an alter- native. Optus had shortened the timer from the industry norm of up to 600 seconds following regulator concerns that some Google Pixel 6A devices running Android 12 could not reconnect to Triple Zero after failed calls. The intention was to allow faster retries. During the outage, the short timer appears to have reduced the likelihood of successful camp-on.
Other Australian carriers use significantly longer settings, typically between 150 and 600 seconds. The review notes that the 10-second timer was the most signiϐicant technical difference between Optus’ network and others in relation to emergency call behaviour. Optus has since reverted the setting to 600 seconds.

Laboratory testing cited in the review shows wide variation in device behaviour. Apple handsets rejected within the 10-second window typically retried via a second IP address and often succeeded. Where there was no response after 10 to 12 seconds, devices began searching for 3G. With 3G shut down, some eventually connected to an- other carrier after 40 to 60 seconds, while others did not.

Samsung devices exhibited different patterns, often attempting multiple strategies simultaneously. Even successful calls frequently took 40 to 60 seconds to connect.
The shutdown of 3G has compounded the issue. Devices that previously relied on 3G fallback now search for a network that no longer exists, introducing delay and uncertainty. Schott suggests the impact of 3G withdrawal on emergency call behaviour may be broader than previously understood.

Optus has begun deeper analysis of device behaviour with support from the University of Technology Sydney, initially focusing on Apple and Samsung handsets. But Schott stressed the issue extends beyond a single telco. “The important issue of the camp-on failure to complete for Triple Zero calls during this outage is most troubling and possibly one for the entire industry and community,” the review concluded, calling for urgent, coordinated investigation.

Grahame Lynch

Schott review details chain of failures behind Optus September outage

A routine firewall upgrade planned weeks in advance and assessed as low risk escalated into a widespread Triple Zero failure because of a cascading series of human errors, missed warning signs and flawed escalation, according to Dr Kerry Schott’s independent review into Optus’ 18 September outage.

The review concluded the outage was not caused by a single technical fault but by “at least ten mistakes” across Optus and its contractor Nokia, stretching from early August planning through to the early hours of 18 September. It finds that established processes existed but were not followed, while inconsistencies that should have triggered escalation were ignored.

Preparation for the upgrade began on 5 August and culminated in work on an Internet Exchange management firewall at the Regency Park exchange in South Australia. Unlike customer-facing demilitarised zone firewalls, IX firewalls do not normally require traffic diversion or isolation, a distinction that proved critical.

The first failure occurred during planning. A change request prepared by Optus and sent to Nokia’s network operations centre in Chennai, India contained incorrect instructions. Internal doubts were raised but not escalated. Despite the upgrade in-volving an IX firewall, an Optus engineer elected to isolate the equipment “to err on the side of caution”. Isolation, however, requires traffic diversion. In this case, the firewall was isolated without traffic being diverted.
When the gateway was locked shortly after midnight on 18 September, ordinary voice calls quickly re-registered on alternative paths. Triple Zero calls did not. About a quarter of emergency callers eventually connected via other routes. The remainder were silently blocked.

The review traced how the decision became embedded. Some network engineers did not attend key meetings in early September. On 8 September, a project briefing was delivered remotely. The engineer involved assumed isolation was required, checked with a colleague and approved the approach on 15 September. The resulting change request instructed Nokia to lock the gateway but did not clearly specify traffic diversion.
This was despite the fact that none of the six previous IX firewall upgrades in the same pro- gram, including one completed on 4 September, had involved isolation or gateway locking. The inconsistency was not escalated.

Errors compounded. On 17 September, Nokia selected a method of procedure dating Nokia’s India facility from 2022 that did not automatically include traffic diversion before gateway locking. A newer procedure, used at the same exchange weeks earlier, was not chosen. The error passed through three peer reviews without detection. Nokia confirmed with Optus that the gateway should be locked and advised work would proceed that night. No response was received.
The work was split across two nights, another unexplained decision highlighted by the review. The gateway was locked on 18 September, with the firewall upgrade scheduled for after midnight on 19 September, leaving the network exposed for around 14 hours. Had the work been completed in a single window, the gateway would have been locked for about six hours.

Risk classification then failed. Nokia assessed the change as both urgent and “no impact”, a combination Schott says was unjustified. As a result, a deeper engineering review was bypassed and the asset was not flagged as high risk, avoiding senior Change Advisory Board scrutiny.

Alerts began firing minutes after work commenced. Nokia dismissed the first alert as related to the planned change. A second alert followed around half an hour later. Optus opened an incident and queried whether it was related to the firewall work, but no investigation followed. Post-implementation checks showed call failure rates rising, not falling, but the anomaly was missed.

Triple Zero monitoring relied on nationwide aggregate data, masking state-based failures. More granular monitoring would have exposed the problem earlier.
Between 10:13am and 2:11pm, five customers contacted Optus reporting they could not reach Triple Zero. Each accurately identified the issue. None were escalated because operators followed a script that focused on immediate danger. One caller even sought advice on purchasing a new handset.

Only after notification from SA Ambulance at 1:31pm did Optus test Triple Zero calls. A major incident was declared at 1:51pm. Nokia was instructed to unlock the gateway at 2:20pm, and service was restored minutes later. By then, 605 Triple Zero calls had failed.

The review concluded warning signs were present for more than 13 hours but were repeatedly dismissed. “The real tragedy,” Schott wrote, “was the duration of the Triple Zero call outage and the failure of Optus and its contractors to deal immediately with the warning signals they received.”

Grahame Lynch

Schott review tells Optus board to harden governance, lift risk discipline and break silos

Dr Kerry Schott’s independent review has warned that Optus’ reform program remains immature and that governance, risk discipline and crisis management fell short of what is expected of an essential service provider.
Schott positioned the Triple Zero outage alongside a series of serious failures in recent years, including the 2022 cyberattack, the 2023 network outage and a court finding of unconscionable conduct linked to sales practices between 2019 and 2023.

Optus’ owner Singtel has responded by shifting away from a matrix reporting structure toward stronger local control, including a strengthened local board and a new CEO reporting to that board. Schott says the transition is incomplete and that the outage “amplifies the need for change and where possible the need to hasten its implementation.”
The reform agenda is framed around strengthening network resilience, simplifying IT complexity and urgently improving processes. Schott notes this involves redirecting investment toward the network, partly at the expense of sales and marketing.

Her most pointed criticism is reserved for risk management. While Optus has adopted the standard three-lines-of-defence model, Schott writes that “it is an understatement to note that the three lines of defence model is nascent at Optus”. The first line of defence failed outright, with processes in Networks at both Optus and Nokia “skipped, ignored or/and not properly checked.” The failure raised questions about the effectiveness of the second and third lines.
Schott said the risk function has historically been too weak, under-resourced and lacking authority. A newly appointed chief security and risk officer now reports directly to the CEO, with a mandate to lift risk capability, maturity and controls assurance, particularly in Networks, Cyber and Regulatory Compliance.

Internal auditing, the third line of defence, also required strengthening. While audits had identiϐied many of the issues highlighted in the review, Schott said the audit function lacked sufficient standing to ensure recommendations were acted upon. The board is now filling key roles, including establishing a chief auditor position.

The review also questioned Optus’ approach to contract management, particularly with Nokia. Schott noted morale in offshore operations appeared poor and argues this may discourage staff from challenging errors. She suggested Optus consider whether a more cooperative partnership model would better support delivery of essential services.

Schott urged Optus to deepen its essential-service culture beyond the frontline. “Telecommunications is now, without doubt, an essential service,” she wrote, citing reliance for payments, medical records and emergency calls.
In crisis response, Schott identified structural weaknesses. Siloed operations delayed escalation to central crisis management, while documentation was unwieldy. One crisis manual exceeded 40 pages. “In a crisis no one is going to examine or read over 40 pages,” she writes.

She said control should have shifted from the Networks unit to central crisis management on the evening of 18 September as the scale of the Triple Zero failure be- came evident. Delays in call centre escalation, including information about potential fatalities, are described as unacceptable.

Grahame Lynch

 

CommsDay over Christmas & New Year
CommsDay’s upcoming Monday 22 December issue will be our last regular publica- tion for the calendar year as we take a publishing break for the summer holidays. We will publish breaking news as required and a summer update around January 5/6. Regular daily publication will resume from 12 January. 

 

IN TODAY'S ISSUE (FRIDAY 19 DECEMBER 2025)

Beyond the operational failures that triggered Optus' 18 September outage, Dr Kerry Schott's independent review released yesterday identified a deeper and unresolved problem: why Triple Zero calls failed to reliably camp on to alternative networks or pathways once the Optus gateway was unavailable. 

A routine firewall upgrade planned weeks in advance and assessed as low risk escalated into a widespread Triple Zero failure because of a cascading series of human errors, missed warning signs and flawed escalation, according to Dr Kerry Schott's independent review into Optus' 18 September outage. 

Dr Kerry Schott's independent review has warned that Optus' reform program remains immature and that governance, risk discipline and crisis management fell short of what is expected of an essential service provider. 

The Australian Competition and Consumer Commission has locked in deep reductions to regulated voice interconnection rates over the next three years, finalising a pricing path that will deliver cumulative cuts of about 70% for fixed termination services and an immediate cut of about 22% for mobile termination, despite strong objections from smaller and fixed-only providers. 

Industry, government and consumer representatives broadly supported the inclusion of copper-based technologies in proposed benchmark service standards for connection and fault rectification reliability, according to a summary of discussions released by the Australian Competition and Consumer Commission. 

The government said it expected its decision to slash the base component of the Regional Broadband Scheme levy by 74% to reduce forecast receipts by $106.8m over three years. 

Freeview Australia plans to begin testing the new Digital Video Broadcasting-Internet technology in Australia from early next year. 

Telstra has extended its satellite messaging service to additional devices including selected Google Pixel phones. 

The New Zealand Telecommunications Forum has revealed that the RE:MOBILE device recycling scheme diverted 71,627 unwanted mobile phones from landfill during FY25.

ST Engineering iDirect and Capgemini have announced a collaboration to develop a 5G non-terrestrial network satellite base station solution. 

 

Plus more

________________________

CommsDay is published by Decisive Publishing, S704 6A Glen St Milsons Point NSW
ACN: 065 084960 Mailing Address: PO Box 490 Milsons Point NSW 1565 Australia

TO SUBMIT EDITORIAL FEEDBACK OR INQUIRIES: grahamelynch@commsdaymail.com
TO ENQUIRE ABOUT A SUBSCRIPTION: vicky@commsdaymail.com

COMMSDAY AND ITS CONTENTS ARE COPYRIGHT. DISTRIBUTION OF COMMSDAY IN YOUR ORGANISATION REQUIRES A SITE LICENCE.
POSTING OF COMMSDAY ON YOUR WEBSITE OR SOCIAL MEDIA IS ILLEGAL AND REQUIRES WRITTEN PERMISSION.