My intrusion detection model scored 0.9998 ROC-AUC. The interesting part was the 19 attacks it missed.<\/p>\n\n\n\n
The assignment was Gradient Boosting on NSL-KDD, a benchmark of 25,192 network connection records. I tuned 27 hyperparameter combinations with five-fold cross-validation. Then I evaluated the winner once on a held-out test set.<\/p>\n\n\n\n
The default settings already reached 0.9996 cross-validated ROC-AUC. The best tuned configuration reached 0.99984. That gain is about the size of the noise between folds.<\/p>\n\n\n\n
An untuned Random Forest then matched the tuned model almost exactly. Only AdaBoost fell behind, with roughly six times the missed intrusions. Its single-split trees couldn\u2019t capture the problem.<\/p>\n\n\n\n
Out of 3,523 intrusions, the model missed 19. None was a scan or a flood. All 19 were TCP, and most were normally completed sessions on ordinary ports: HTTP, FTP, telnet, SMTP.<\/p>\n\n\n\n
Scans and floods are easy. They leave structural fingerprints, like a SYN with no reply. Signature-based tools already catch those.<\/p>\n\n\n\n
The missed attacks looked like normal sessions carrying a malicious payload. Those are exactly the cases machine learning is supposed to add value against.<\/p>\n\n\n\n
It wasn\u2019t a threshold problem either. Most of the misses scored below 0.11 probability of being an attack. The model was confidently wrong.<\/p>\n\n\n\n
One feature, source bytes, carried 72.8 percent of the model\u2019s importance. A model leaning that hard on payload size separates floods easily. It has little to say about a telnet session carrying a few hundred bytes of password guesses.<\/p>\n\n\n\n
The near-perfect score described the easy attacks. The useful result was the structure of the errors left over.<\/p>\n\n\n\n
The full report is on my CYSE 420 course page<\/a>.<\/p>\n\n\n\n