Network Security Monitoring

1. What is Network Security Monitoring?

NSM is the continuous collection and analysis of network traces — packets, flows, and logs — to detect abnormal behaviour and support incident response. Where EDR tools watch what happens on an endpoint, NSM watches what happens between endpoints. The network, unlike a compromised host, cannot lie to you.

NSM is built around answering four foundational questions about any suspicious activity. If you can answer all four, you have enough to scope the threat and begin response.

Who?

Identity

Source & destination IPs, resolved hostnames, and associated user accounts.

When?

Timeline

Exact timestamps, connection frequency, and periodicity patterns across time.

How?

Protocol & Port

Which protocols were used (DNS / HTTP / SMB) and over which port numbers.

How much?

Volume

Bytes in / out, file sizes, connection count — data volume often reveals intent.

2. The Three Types of Network Evidence

Not all network data is equal. Each evidence type offers a different trade-off between detail and speed. Experienced analysts know when to pull which type — using the wrong one wastes time and can miss the signal entirely.

Type	Detail Level	Best For	Analogy
Packet (PCAP)	Extremely High	Deep forensics — seeing exactly what was inside the "envelope." Every byte of every payload.	Opening the letter and reading every word.
Flow	Summary	Fast threat hunting — tracking connections, ports, duration, and volume at scale without storing full payloads.	Reading the envelope: sender, recipient, weight, time.
Metadata / Logs	Contextual	Quick pivots — inspecting specific HTTP requests, DNS queries, or TLS negotiation details.	The post office's delivery manifest — structured, organised, searchable.

When to use what: Start with Flows and Metadata/Logs to identify the suspicious host and timeframe (fast). Then pull the PCAP only for the specific connection you've already scoped (targeted). Pulling full PCAPs first is slow, storage-heavy, and noisy.

3. NSM Tools Landscape

NSM has a well-established ecosystem split between open-source powerhouses and commercial platforms that typically build on top of them. Understanding both sides matters when you're working in enterprise environments where either may be deployed.

Category	Tool	Key Strength
Open-Source	Zeek (formerly Bro)	Interprets raw traffic into structured, protocol-aware logs. The foundation of modern NSM.
Open-Source	Suricata	High-performance IDS/IPS with signature-based detection. Runs rules against live traffic.
Open-Source	Snort	The original open-source IDS. Widely supported ruleset ecosystem via Snort/Talos rules.
Commercial	Corelight	Enterprise appliance built on Zeek — same logs, hardened hardware, centralised management.
Commercial	Palo Alto Networks	Next-gen firewall with integrated App-ID, threat prevention, and DNS security.
Commercial	Fortinet	NGFW + UTM suite covering IDS, web filtering, and sandboxing in a unified platform.

4. Deep Dive — Zeek Logs

Zeek doesn't just record traffic — it interprets it. Rather than storing raw packets, Zeek parses each protocol and writes structured, query-ready log files. This is what makes it the foundation of modern NSM pipelines.

conn.log

The Connection Log

The backbone of NSM. Records every completed network connection — who talked to whom, for how long, and how much data moved in each direction.

Beaconing & Port Scan Detection

ts Timestamp of the connection start
id.orig_h Originating (source) IP address
id.resp_h Responding (destination) IP address
duration How long the connection lasted
orig_bytes Data volume sent by the source
resp_bytes Data volume sent by the destination
history TCP handshake story (e.g. ShAdDaf)

dns.log

The Domain Log

Captures every DNS query and response. DNS is the phone book of the internet — and attackers abuse it constantly to reach C2 infrastructure or exfiltrate data.

DNS Tunneling & DGA Detection

query The domain name being looked up
qtype_name Record type (A, AAAA, TXT, MX…)
answers The resolved IPs or CNAME returned
rcode_name Response code — watch for NXDOMAIN

Red Flags

NXDOMAIN storms — DGA malware cycling through generated domains
Strange TLDs — .onion in a corporate log is a massive red flag
Long subdomains — 60+ char queries often indicate DNS tunneling

http.log

The Web Log

Records every HTTP transaction — host, URI, request method, response code, and user agent. Essential for tracking payload downloads and web-based C2 communication.

Payload Downloads & Web C2

host The HTTP Host header value
uri The requested path and query string
method GET / POST / PUT (POST = data exfil risk)
user_agent Browser/tool string — malware often fakes or omits this
resp_mime_types File type served — watch for .exe, .ps1

ssl.log

The TLS/SSL Log

Captures TLS handshake metadata for encrypted traffic. Even though you can't read the payload, you can fingerprint the client from how it negotiates encryption.

Encrypted C2 & JA3 Fingerprinting

server_name SNI field — domain inside the TLS handshake
ja3 Client TLS fingerprint hash
ja3s Server TLS response fingerprint
validation_status Certificate validity state

The `history` Field — Reading TCP Stories

The history field in conn.log encodes the full lifecycle of a TCP connection as a sequence of letters. Learning to read it is one of the fastest ways to spot scanning, stealthy connections, and abnormal session teardowns without needing a PCAP.

                    S → SYN sent # Connection initiated
                        by client

                    h → SYN-ACK received # Server
                        acknowledged — port is open

                    A → ACK sent # Three-way handshake
                        complete

                    d → Data transferred # Payload
                        bytes exchanged

                    f → FIN sent # Normal, clean
                        session close

                    R → RST received # Abrupt reset —
                        port closed or filtered


                    # Port scan signature: many connections with just 'S' (no reply)

                    history = "S" → SYN sent, NO SYN-ACK = closed/filtered
                        port

                    # Hundreds of these from one IP = active port scan in progress
                

Beaconing memory trick: Look at the ts field in conn.log. If a connection recurs at an unnervingly consistent interval — say exactly every 5 minutes (08:01, 08:06, 08:11…) — that regularity is the signature. Humans don't communicate on rigid schedules. Malware does.

5. Strengths, Limitations & Pro Techniques

Strengths

Detects threats even when the endpoint is compromised and "lying" to your EDR
Provides neutral, tamper-resistant evidence — the network doesn't lie
Defines the "Blast Radius" — how far a threat spread across the environment
Effective against lateral movement, exfiltration, and C2 communication
Retroactive analysis — stored logs let you investigate past timeframes

Limitations

TLS Encryption: Can see who is talking, but not what they are saying
No Process Data: Cannot identify which .exe initiated a connection — need EDR for that
Storage Cost: Full PCAP retention at scale is extremely storage-intensive
Blind to Host: In-memory malware that never hits the wire is invisible

Pro Hunting Techniques

Technique 01

UID Correlation — The Holy Grail

Every single connection across all Zeek logs shares a uid field. Found a suspicious domain in dns.log? Grab its UID and grep every other log for it. You'll instantly see the conn.log entry (data volume) and any http.log request (what was downloaded).

grep "CH9p9b266YvXqT847" *.log
# Links dns → conn → http for a single event

Technique 02

Beaconing Detection with Standard Deviation

Malware beacons home at regular intervals. Humans are never that consistent. Calculate the standard deviation of time intervals between connections to a single external IP. Near-zero jitter = automated. Use RITA (Real Intelligence Threat Analytics) to automate this at scale.

# What you're looking for:
stdev(intervals) → ~0 ← Malware beacon
stdev(intervals) → high ← Human / random

Technique 03

DNS Tunneling — Query Length Analysis

Attackers encode exfiltrated data inside DNS query strings to bypass firewalls. Normal domain queries are short. Tunneled queries are extremely long. Sort your dns.log by query length descending and anything over ~60 characters warrants scrutiny.

# Normal DNS query:
google.com (10 chars)

# Tunneling signature:
v3a4f1b2...topsecret.evil.com (100+ chars)

Technique 04

JA3 Fingerprinting for Encrypted C2

When TLS hides the payload, JA3 fingerprints the process. A JA3 hash is derived from how the client negotiates TLS — cipher suites, extensions, elliptic curves. Even if malware changes its C2 IP or domain, its JA3 hash typically stays constant. Known-bad hashes (e.g. Cobalt Strike) are publicly listed.

# In ssl.log:
ja3 = "72a7c94f55441fac..."
# Cross-reference with threat intel feeds

Technique 05

NXDOMAIN Storms — DGA Hunting

Domain Generation Algorithms (DGAs) make malware resilient — they algorithmically generate hundreds of domain names and try each until one resolves to an active C2. The symptom in dns.log: a burst of NXDOMAIN responses from a single host, often with pseudo-random looking domain names.

# Hunt: high NXDOMAIN rate per source IP
rcode_name == "NXDOMAIN"
# Group by orig_h, sort by count desc

Further reading: The official Zeek Quickstart Guide covers UID correlation in depth. For automated beaconing detection, Active Countermeasures' RITA is the go-to open-source tool. JA3 hashes can be validated against the Salesforce Engineering JA3 GitHub repository.

Key Takeaways

NSM answers the four questions every analyst needs: Who, When, How, and How much.
Start with Flows/Metadata to scope fast — pull PCAPs only for targeted, confirmed leads.
Zeek's UID is the analyst's superpower: one hash links a suspicious event across every log type simultaneously.
The history field in conn.log tells the complete TCP story — a string of S with no reply is a port scan.
Beaconing looks like inhuman regularity — near-zero standard deviation in connection intervals.
DNS is not just a lookup protocol for attackers — watch for NXDOMAIN storms (DGA) and long query strings (tunneling).
JA3 fingerprints let you identify malware families in encrypted traffic without ever seeing the payload.
NSM and EDR are complementary — the network sees what the endpoint hides, and the endpoint sees what the network can't.

Network Security Monitoring (NSM)