My intrusion detection model scored 0.9998 ROC-AUC. The interesting part was the 19 attacks it missed.
The assignment was Gradient Boosting on NSL-KDD, a benchmark of 25,192 network connection records. I tuned 27 hyperparameter combinations with five-fold cross-validation. Then I evaluated the winner once on a held-out test set.
Tuning barely mattered
The default settings already reached 0.9996 cross-validated ROC-AUC. The best tuned configuration reached 0.99984. That gain is about the size of the noise between folds.
An untuned Random Forest then matched the tuned model almost exactly. Only AdaBoost fell behind, with roughly six times the missed intrusions. Its single-split trees couldn’t capture the problem.
What the misses had in common
Out of 3,523 intrusions, the model missed 19. None was a scan or a flood. All 19 were TCP, and most were normally completed sessions on ordinary ports: HTTP, FTP, telnet, SMTP.
Scans and floods are easy. They leave structural fingerprints, like a SYN with no reply. Signature-based tools already catch those.
The missed attacks looked like normal sessions carrying a malicious payload. Those are exactly the cases machine learning is supposed to add value against.
It wasn’t a threshold problem either. Most of the misses scored below 0.11 probability of being an attack. The model was confidently wrong.
Why
One feature, source bytes, carried 72.8 percent of the model’s importance. A model leaning that hard on payload size separates floods easily. It has little to say about a telnet session carrying a few hundred bytes of password guesses.
The takeaway
The near-perfect score described the easy attacks. The useful result was the structure of the errors left over.
The full report is on my CYSE 420 course page.