The Six Hours Our Signatures Missed
A zero-day exploit against a web application resulted in outbound connections to 14 new IPs over 6 hours, and our signature tools detected nothing. I saw the pattern later in raw flow records from our manufacturing DMZ, after the incident bridge had filled with web logs and proxy searches.
My first assumption was wrong. I thought one of our perimeter tools had failed to update or that a FortiOS 7.4.3 IPS profile had been applied to the wrong policy. The real problem was simpler and worse: we had no behavioral baseline to detect the anomalous outbound connection pattern; our SIEM was alert-only for known signatures.
The exploited application did not spray malware hashes across the network. It did not trip a known command-and-control domain. It made low-volume outbound connections to new destinations, slowly enough to look uninteresting to tools waiting for a named indicator.
Nothing blinked.
We run Python 3.11 collectors on Ubuntu 22.04 for several operational feeds, but at that point NetFlow was mostly a troubleshooting source, not a security detection source. My opinion is that zero-day detection starts when I stop asking whether traffic matches yesterday’s threat and start asking whether it belongs in my environment.
Baseline Normal Traffic Before the Incident
NetFlow and IPFIX gave us the data we should have been using earlier: source, destination, ports, bytes, packets, duration, and timing. That was enough to describe normal behavior for our web tier without inspecting payloads or waiting for signatures. I wanted to know which servers talked outbound, how often, to which countries, on which ports, and during which production windows.
We built baselines by role, not by subnet alone. A public web server, a historian collector, an engineering workstation, and an Ubuntu 22.04 jump host do not share the same normal. Grouping them together makes every threshold either too noisy or too blind.
The useful baseline was boring: top destinations, new destination rate, failed connection ratio, byte symmetry, connection duration, and time-of-day profile. When a web server that normally reached 3 external update services suddenly touched 14 new IPs in 6 hours, that should have been enough to start asking hard questions.
Normal is evidence.
After implementing NetFlow-based behavioral analysis, the post-incident simulation detected the same traffic pattern in 4 minutes with a behavioral alert. My opinion is that this kind of detection is not fancy; it is the minimum math we owe ourselves before the next unknown exploit lands.
Look for Movement Without Waiting for Names
Zero-day activity often becomes visible after the initial exploit, when the attacker tries to understand the environment. In a plant network, that movement can be subtle because legitimate systems also talk across zones for scheduling, quality checks, badge access, and maintenance data. I cannot rely on a generic lateral movement signature to understand that context.
We started scoring lateral behavior by comparing each host to its own history and peer group. A web application server opening SMB connections into an engineering VLAN is different from a domain controller doing the same thing. A maintenance workstation scanning high ports during a planned vendor session is different from a service account host doing it at 02:13.
- I flag new east-west destination pairs for production-facing servers.
- I watch for first-time administrative ports from non-admin host roles.
- I score connection fan-out when one host touches many peers quickly.
- I compare byte ratios to catch probing, staging, and small command traffic.
- I tag flows with asset role so alerts reflect operational context.
What I didn’t expect was how often the best signals were small. The post-exploit flow did not look like a dramatic scan. It looked like a server becoming curious about places it had never needed before.
Curiosity is suspicious.
My opinion is that lateral detection gets stronger when it respects the plant’s normal work instead of flattening every subnet into the same rulebook.
You may also find this useful: Check out our guide on Python Network Config Backup: Automating Multi-Vendor Device Snapshots for more practical tips.
Catch Beaconing by Measuring Rhythm
Command-and-control traffic rarely announces itself by name during a novel attack. I look for rhythm instead: regular intervals, repeated small payloads, unusual persistence, and destination changes that stay within the same hosting pattern. A domain blocklist cannot help when the infrastructure is new, but timing still leaks intent.
Our first beaconing detector was intentionally plain. We grouped flows by source, destination, and destination port, then calculated interval consistency and outbound byte size variance. If a server contacted a new IP every 300 seconds with nearly identical packet counts, I wanted a human to see it even when no vendor called it malicious.
awk -F, '{print $1,$2,$5,$7,$8}' flows.csv \
| sort \
| python3.11 detect_beacon_intervals.py \
--min-events 6 \
--max-jitter 20 \
--new-destination-window 7dThat script did not replace the SIEM. It produced a compact set of behavioral findings with source host, destination, port, interval, jitter, and first-seen time. Then we forwarded those findings into the same queue my analysts already used.
Rhythm leaves tracks.
I do not believe every periodic connection is evil, and our manufacturing environment has plenty of scheduled traffic. My opinion is that beacon detection earns its keep when it combines timing with novelty, role context, and destination reputation instead of shouting about every heartbeat.
Integrate Behavioral Alerts Into the Analyst Workflow
The fastest way to waste a good detector is to make it live outside the workflow. We pushed NetFlow behavioral alerts into our SIEM with fields analysts already used: host owner, asset role, source zone, destination ASN, first-seen timestamp, baseline deviation, and raw flow sample. I wanted one-screen explanation.
We also tuned severity differently from signature alerts. A known exploit signature on an internet-facing system can page immediately. A behavioral deviation starts as triage unless it stacks with other evidence, such as new outbound destinations plus odd lateral connections plus a fresh web error spike. That kept trust high because analysts saw fewer single-signal emergencies.
Every alert included a question, not just a label. Did this server talk to this destination before? Does this destination match an approved vendor? Did the timing start after a deployment? Is there a change record? Those questions made the alert useful even before attribution was clear.
Context keeps alerts alive.
I still run signature tools, IPS profiles, EDR, and proxy controls because known threats deserve known detections. I just no longer pretend those tools cover the unknown. My opinion is that zero-day readiness comes from watching behavior early enough that the absence of a signature does not become the absence of action.