Common Firewall Log Analysis Mistakes and How to Avoid Them
It’s 3rd coffee hour here at my desk—and if you’ve ever had to look through a firewall log, you know it’s like finding a needle in a haystack. But after all, spending close to 3 decades in networking and cybersecurity (I did start out as a network admin in 1993 – yes, analog phone days, amm) I’ve learned a couple of lessons about what gets people when they monitor firewall logs for security incidents.
Here’s the deal: firewall logs are a goldmine of information – but not if you fall into these firewall logging pitfalls. I’ve assisted numerous clients (yes, three banks recently upgraded their zero-trust architecture — debilitating but fruitful s**t) and audited numerous firewall setups at PJ Networks Pvt Ltd. So, pull up a chair.
Common Firewall Log Analysis Mistakes and Solutions
Let’s dissect:
- Type or category of Firewall Log Analysis mistakes
- Degree of the impact
- Location of a device and source of activity
- System specific aspects
- Different Firewall Log Settings
- Time they have been acting and the data
- Motion on the network
1) Unfiltered Logs
Impact zone: Infected
What you need Done yet how/where is how you will present or deliver it
2) Delete logs frequently from disk
Impact zone: Infected
Do some kind of investigation
3) Assuming errors are logged
Impact zone: Infected
Overviews and summaries
4) Overuse of Alert
Impact zone: Trainees
Overviews and summaries
5) Configuration – OS Firewalls only
Impact zone: SOHO – small office/home office, dorm or apartment
Network Discovery and Download
6) Configuration – “allow all”
Impact zone: C & C address
What’s next? Ignoring Low-Level Alerts
You’d think ignoring low-level alerts is beginner-level fodder — but you would not believe the numbers of seasoned veterans who dismiss these. Cut to the early 2000s when I just got started watching the Slammer worm and it was proliferating like mad— emanating from what appeared to be innocent, inconsequential packets nestled deep in logs.
But such low-level alerts usually provide the first signs of an attack. Ignoring them would be like ignoring the first coffee indicator of engine trouble in your car; it might not be a catastrophe right away — but it’s headed that way.
Here’s what I’ve seen:
- Swamps of alert rain labeled INFO or NOTICE that are ignored because they are not critical
- Infrequently related minor errors or unexpected connect attempts
- Because you’re programmed like a YouTube passenger in rush hour = no prob
Don’t do that. Alternatively: look for patterns within and for combinations of these low-level signals. They are often the harbingers of bigger attacks or reconnaissance.
Pro Tip: Design your alerting so that you log these alerts from above with retention, as well as alert up(cascading) if you notice something unusual.
Failing to Correlate Logs
Oh boy. Log correlation — this is where a lot of us firewall admins get caught. Logs in isolation? Pretty much useless. But put them together? A map of ingress points and trespasser traces.
Old school here — I remember when log correlation meant piecing together events from your routers, firewalls, and servers by hand. Today, sure, we have fancy SIEMs, but blindly trusting them? Not the move.
Here’s what often goes wrong:
- Viewing firewall logs in isolation and not combining them with VPN, IDS, authentication and endpoint logs
- Not reconciling timestamp discrepancies between columns
- Dismissing canaries that appear harmless independently but dangerous when considered together
Because here’s the reality: An attacker pings your firewall, then they move to an endpoint. Firewall log indicates that a connection has been allowed. Endpoint log is visible failed logins. Alone, meh. Together? Major red flags.
The upshot: If you’re not correlating, you may as well be working blindfolded.
How to fix it?
- Consolidate disparate logs into a single source so you can write correlating rules for log entries
- Strictly practice time synchronization (NTP or similar) for all your devices – no compromises
- Train your staff to look for signal failures as part of the singly signaled alarm side effects
Poor Retention Policies
Some days I want to scream this from the rooftops: Logs are a waste if they disappear when you wish they wouldn’t.
I have audited firewalls with retention set for days, not weeks, not months. Memory is cheap, people — let’s not pretend otherwise. If an attacker is clever, he’ll attempt slow, surreptitious reconnaissance that can take weeks. Throw your logs early, and you’ll never trail their scent.
A bank client of mine, for instance, demanded a 7-day retention policy “to save space.” Their logs disappeared after a breach. Guess who might have stitched the timeline of the attack? Yeah.
Brief rant alert: If anyone is griping about log storage costs in Q1 2024, they deserve a slap. You’re protecting data, not a collection of cat videos.
Some rules of thumb:
- Retain at least 90 days of firewall logs in the course of business.connections.
- Save older logs in a secure place at the archive level — and encrypt them
- Consider using automated backups as part of your Log Retention strategy
Not Using Threat Intelligence
Okay, this is where the takes might get a little controversial. There are lots of people around me who are chasing AI powered firewalls. I mean, don’t get me wrong, I’m not anti-AI, but I am skeptical of black boxes that can enrichen logs automatically and “unassailably” find danger without any human intervention.
Threat intelligence feeds—if implemented right—are pure gold. But if you’re just letting your firewall “auto-update” intelligence without looking at what it’s catching, that’s a problem.
Here are a few of the dumbest misses I’ve encountered:
- Not updating threat feeds
- Failure to tailor threat intel for your environment
- Blindly accepting generic world-wide threat databases as the gospel truth (or the infallible guide)
Threat intel is supposed to provide context, not just alerts. Consider cooking — if you throw in every spice to a broth with no measurement, you’re making garbage. Personalize feeds for your industry, geo-locations, and past.
My take: Leverage threat intelligence to supplement firewall logs analysis. But give that gut feeling some environment-specific findings and yes—your manual review.
Lack of Real-Time Monitoring
I understand, real-time monitoring can be resource-intensive. But compromises are easier to detect than they are to prevent and, if you’re checking firewall logs days after a potential threat is mitigated, I’ve got news for you: You may have already been breached.
We always advise firewall specific monitoring dashboards at PJ Networks.
When I was young the way you did monitoring was your entire team gathered round a bank of (literally) green and red blinky lights. Now — it’s dashboards with alerts that trigger when things go off-kilter. Even so, some businesses continue to audit logs manually, occasionally. That’s a recipe for disaster.
Things that help:
- Firewall integrated real-time alerting tools
- For high priority incidents, SMS or push notifications
- Constantly re-tuning of alert thresholds to not become over sensitive
During my recent travels (returned from DefCon recently and am still thinking way too much about the hardware hacking village), I saw how attackers are able to launch second-long exploits. If you don’t intercept oddball traffic on your way in, well, your firewall is essentially a welcome mat.
Quick Take
- Low-level alarms are important—don’t dismiss them.
- Correlate system logs between systems and the whole story is revealed.
- Retain your logs for long enough that you can analyze slow attacks.
- Threat intelligence is a weapon, not a silver bullet.
- Monitor real time or be too late.
TL;DR? Firewall log data is much like cooking a complex curry—let one component go and the whole thing collapses. Pay attention to each byte and you just might protect your network from that ugly zero-day we protected those three banks from recently.
I’m Sanjay Seth from PJ Networks reminding you, do not put your firewall logs on the shelf to collect dust like old parchments. Dive in, correlate, suck it up and watch for the (youpickings) — because the bad guys sure do.
And hey — next coffee’s on. Perhaps next time I’ll write about password policy madness. Spoiler: Complexity should not be confused with security.
Stay safe,
Sanjay