Neural Network IDS vs. Tree Ensembles on NSL-KDD

I expected the neural network to win. It didn’t.

This assignment was a multi-layer perceptron (MLP) intrusion detector built in TensorFlow/Keras on NSL-KDD network traffic. I used the same test set as my earlier Gradient Boosting assignment, so the comparison was direct.

Same detection, six times the alerts

The MLP missed 20 intrusions. Gradient Boosting missed 19, and Random Forest missed 18. On detection, that’s a tie.

False alarms were a different story: 27 for the MLP, against 5 and 4 for the trees. That’s six times the alerts for the same detection rate. In a security operations center, it’s the difference between a tool analysts trust and one they learn to ignore.

More work for a worse result

The trees needed no feature scaling. The MLP needed log transforms, standardization, a validation split, early stopping, and a learning-rate search.

That fits a broader pattern. Neural networks shine on data with structure, like images or sequences. Network connection records are a table of independent features, so there’s no structure to exploit.

Where the errors lived

Over half the false alarms were ordinary ping traffic. In this dataset, ICMP echo also carries smurf attacks, and the features can’t tell the two apart.

Fifteen missed intrusions scored near zero. No threshold adjustment would recover them.

An honest note on tuning

Tuning showed that learning rate mattered and batch size didn’t. But the top four settings sat closer together than the noise from rerunning a single one. Tuning didn’t produce a measurably better model.

The takeaway

The newer model isn’t automatically the better one. On tabular security data, the tree ensembles were simpler and more accurate.

The full report is on my CYSE 420 course page.