A spam filter that never catches a single spam message would still score 87.4 percent accuracy on my dataset. That one fact shaped this whole assignment.
I built two classifiers for SMS spam: logistic regression and multinomial Naive Bayes. I trained each on two train/test splits, 70/30 and 90/10, and compared them on five metrics.
Why accuracy failed
After removing 403 duplicate messages, only 12.6 percent of the data was spam. Labeling everything as legitimate gets you 87.4 percent for free.
All four of my runs landed between 97.2 and 98.5 percent accuracy. By that measure, they were basically equal.
What recall showed
They weren’t equal. On the 70/30 split, Naive Bayes caught 91 percent of spam and missed 18 messages. Logistic regression missed 42.
Logistic regression was the cautious model. It flagged only one legitimate message as spam, but it let much more spam through. Naive Bayes struck the better balance, with the higher F1 score on both splits.
Which one is “better” depends on the cost of each mistake. A false positive hides a real message. A false negative puts spam in the inbox.
The threshold, not the model
Both models scored 0.98 to 0.99 ROC-AUC. That means they ranked spam above legitimate messages almost equally well. Logistic regression’s low recall came from the default 0.5 cutoff, not from weak learning. Lowering its threshold would recover much of the gap.
The takeaway
On imbalanced data, the metric you pick decides what you see. Accuracy said these filters were the same. Recall said they weren’t.
Other related coursework CYSE 420 course page.