The Monday Shift Change That Exposed Our MFA Blind Spot
MFA push notifications stopped delivering after an iOS 17 update, and our operators quietly switched to TOTP codes without telling IT. I found out at 06:42, not from an alert, but from a maintenance planner who asked why FortiToken Mobile had become “slower than usual” after his phone update. By then, our FortiGate VPN running FortiOS 7.4.3 was still enforcing MFA, but the push path was dead.
My first assumption was wrong. I blamed iOS 17 because the timing looked perfect, and I spent the first 25 minutes comparing device settings, notification permissions, focus modes, and FortiToken Mobile app versions. The real issue was uglier: the FortiAuthenticator APNs certificate had expired 3 days before the iOS update, and the update only exposed a pre-existing failure that our team should have caught earlier.
Silent failure is the worst failure.
Our environment was not exotic. We had FortiAuthenticator 6.6.2 handling token assignment, FortiGate 200F appliances on FortiOS 7.4.3 terminating SSL VPN, Ubuntu 22.04 jump boxes for log review, and a Python 3.11 script that pulled authentication events into our internal reporting stack. The part that bothered me was not that something expired. Certificates expire. The problem was that push degradation looked like user choice instead of infrastructure failure, and I think that is a dangerous default.
Trace FortiToken Push Delivery From Login To Phone
When one of our users starts an SSL VPN login, the FortiGate validates the primary credential and then asks FortiAuthenticator to satisfy the second factor. For a push attempt, FortiAuthenticator does not magically reach into the phone. It sends a push request through the Fortinet token service path, which depends on Apple APNs for iOS devices and Google FCM for Android devices. The phone receives the notification, the FortiToken Mobile app opens the approval flow, and the response travels back through the FortiToken ecosystem before FortiAuthenticator tells FortiGate whether to allow the session.
That chain gave me four places to look: the FortiGate authentication request, the FortiAuthenticator token state, the outbound firewall path, and the mobile notification provider dependency. I had primary authentication working, TOTP working, and VPN authorization working, so the broken segment had to sit between FortiAuthenticator and mobile push delivery.
The password was never the problem.
I checked the obvious FortiOS side first because our VPN team lives there every day. These commands helped confirm that FortiGate was still handing MFA to FortiAuthenticator cleanly and that the VPN policy path was not dropping users after primary authentication.
diagnose debug reset
diagnose debug application fnbamd -1
diagnose debug application sslvpn -1
diagnose debug enable
config user radius
edit "FAC-MFA"
show
next
end
diagnose test authserver radius FAC-MFA j.smith 'temporary-test-password'Those traces showed the request leaving FortiGate and the FortiAuthenticator decision path responding. I do not like troubleshooting MFA from only the firewall side because it creates a false sense of control; push MFA is partly outside the firewall, and pretending otherwise wastes time.
Check FortiAuthenticator APNs And FCM Before Blaming Phones
On FortiAuthenticator, I moved straight into the mobile token and system certificate views. The APNs certificate showed expired, and the timestamp was 3 days before the first iOS 17 help desk note. That was the moment the story changed from “Apple broke push” to “our monitoring missed a certificate lifecycle event.” I renewed the APNs certificate, validated the FortiToken Mobile service status, and checked that Android users still had a clean FCM path.
What I didn’t expect was how normal the failure looked from the user side. Nobody reported “MFA is down.” They reported “I used the code instead,” which sounded like a preference, not a production incident. In a manufacturing facility, that distinction matters because shift workers will pick the fastest path that gets them into the maintenance portal, MES dashboard, or vendor VPN session.
Convenience hid the outage.
- I now check APNs certificate expiry during every monthly access-control review.
- We track FCM reachability separately because Android success can mask iOS failure.
- We alert on push-to-TOTP ratio changes, not just failed logins.
- We review FortiAuthenticator system events after every mobile OS release window.
- We test with managed and unmanaged phones because notification behavior differs.
After renewal, push delivery restored to 99.4% within 2 hours of certificate renewal; we had zero user-impacting failures in the subsequent 90 days. Before renewal, our practical push delivery rate for affected iOS users was 0 pushes, because nothing reached the device. I trust metrics that compare behavior before and after a fix more than I trust a green dashboard.
Allow The Firewall Paths Push MFA Actually Needs
Our FortiAuthenticator sits in a restricted server VLAN, so outbound access is intentionally narrow. That is the right posture for a security appliance, but narrow egress becomes a problem when the appliance depends on external notification providers. I verified DNS resolution, TLS inspection exemptions, proxy behavior, and destination access for Fortinet services, Apple APNs, and Google FCM. I also checked that our FortiGate policy did not force certificate inspection on traffic that needed clean provider TLS negotiation.
In our case, the certificate expiry was the primary cause, but I found an old firewall rule that allowed the previous FortiAuthenticator address object and not the current HA pair address. That rule had not caused this incident, but it would have caused the next one after failover. I fixed it during the same change window because leaving a known sharp edge in MFA infrastructure is bad engineering.
You may also find this useful: Check out our guide on Python Network Config Backup: Automating Multi-Vendor Device Snapshots for more practical tips.
Near misses count.
For our environment, I keep the egress rule boring and explicit: FortiAuthenticator outbound to required Fortinet token services, APNs, FCM, DNS, and NTP, with logging enabled and no TLS break-and-inspect on those flows. I also pin the rule comments to the ticket that approved the dependency, because six months later somebody will ask why a security system needs internet egress. My opinion is simple: MFA infrastructure deserves stricter review, not magical thinking that external dependencies do not exist.
Detect TOTP Fallback Before It Becomes Normal
TOTP saved us that morning, and I am glad it did. The problem is that fallback can turn into camouflage. If users can complete authentication with a six-digit code, the business process keeps moving while push MFA quietly rots behind it. I had to stop treating successful MFA as a single bucket and start separating push approval, TOTP fallback, push timeout, and user cancellation.
Our Python 3.11 reporting job on Ubuntu 22.04 now calculates a rolling push success baseline from FortiAuthenticator logs and raises an internal alert when TOTP usage jumps above our expected range. I care less about one user choosing a code and more about a plant-wide pattern that starts within one shift. We also added a daily synthetic test using an IT-owned iPhone on iOS 17.4 and an Android 14 device so APNs and FCM both get exercised.
One token path is not enough.
The best signal came from ratios. When push attempts dropped and TOTP completions rose at the same time, the system was still secure enough to block unauthenticated access, but the user experience had degraded. That is exactly the kind of failure that does not scream in a NOC view. I now believe fallback telemetry should be treated as availability monitoring, not just authentication reporting.
Renew Certificates Before Mobile Updates Take The Blame
I changed our operating rhythm after this incident. We added FortiAuthenticator certificate expiry checks to the same calendar we use for SSL VPN certificates, FortiOS upgrade planning, and quarterly access reviews. We also created a pre-mobile-release checklist because iOS and Android updates tend to get blamed for anything that happens nearby, even when the root cause is older and quieter.
I still think iOS 17 mattered, but not because it broke FortiToken Mobile. It changed user behavior enough for us to notice a failure we already owned. That distinction made our fix better: renew the APNs certificate, validate FCM, clean the firewall egress rule, measure push delivery directly, and stop accepting TOTP growth as harmless background noise.
Blame is not a root cause.
My strongest opinion from the whole event is that MFA push should be monitored like a production dependency, not treated like a phone feature. If APNs, FCM, FortiAuthenticator certificates, DNS, NTP, or firewall egress break, users will often keep working around the issue until the workaround becomes normal. I do not want normal to mean invisible degradation.