DNS Tunneling Detection: ML Pipeline

A machine learning pipeline that detects DNS tunneling, the covert channel that smuggles data out inside ordinary DNS queries. It learns how queries are shaped and how a source behaves over time instead of matching tool signatures, and it holds up on captures it has never opened: 99.99% detection on unseen tunneling tools and 99.84% across a different endpoint.

Project Overview

DNS is the one protocol nearly every firewall lets through, which makes it the quietest way out of a network. Tunneling abuses exactly that: data is encoded into the subdomain of a query, a server the attacker controls decodes it, and the exfiltration looks like ordinary name resolution. Signature-based detection catches the tools people already know about and misses the next one, so this pipeline takes the other route and learns the channel itself.

Every packet becomes 34 features. 17 lexical ones read the query string (Shannon entropy on the full query and on the subdomain, length, label depth, digit, uppercase, base64 and hex ratios), 14 behavioral ones are computed over a rolling 200-packet window grouped by source IP and session (query rate, inter-arrival and packet-length statistics), and 3 flag duplicates and retransmissions. Four classifiers were trained and compared on a labeled PCAP corpus split at the file level rather than the row level, so the test set is made of captures the model has never opened instead of shuffled rows from the same traffic burst.

It was then pushed against data kept strictly out of training: tunneling tools absent from the training set (tcp-over-dns, Cobalt Strike, dns2tcp, OzymanDNS), captures recorded on a different endpoint (AndIodine on Android), and legitimate wildcard DNS traffic held out as a false-positive trap. Detection held at 99.99% on the unseen tools and 99.84% across endpoints, which is the evidence that the model learned the structure of the channel rather than the fingerprints of the five programs it was trained on.

Key Features

34 features per DNS packet: 17 lexical (Shannon entropy on the query and the subdomain, length, label depth, digit, uppercase, base64 and hex ratios), 14 behavioral (query rate, inter-arrival and packet-length statistics), and 3 duplicate and retransmission flags
Behavioral features computed over rolling 200-packet windows grouped by source IP and session, so the model reads a source's rhythm and not just one query in isolation
File-level train and test split instead of a row-level one, which removes the leakage that silently inflates results when near-duplicate packets from the same traffic burst land on both sides
Four classifiers trained and compared (Logistic Regression, Random Forest, XGBoost, LightGBM), reaching roughly 1.00 ROC-AUC and F1 on the in-distribution test set
Leave-one-tool-out evaluation across the five training families (dnscat2, dnspot, iodine, DNS-shell, tuns): the model is retrained with one family removed and tested only on that family, so generalization is measured rather than assumed
Three stress sets kept strictly out of training: unseen tunneling tools at 99.99% detection, captures from a different endpoint at 99.84%, and legitimate wildcard DNS held out as a false-positive trap