My Model Hit 85% Accuracy. Here's What That Number Is Hiding.

When I finished the Mental Health Prediction App and saw 85%+ accuracy on the test set, I felt good about it. It went into the project README as a headline metric. I put it on the portfolio card.

The more I sat with it, the less comfortable I became. 85% accuracy sounds like a strong result. In the context of predicting mental health risk from survey data, it might be masking the most important failure modes almost entirely.

This is what I should have written in the README instead.

What 85% accuracy actually tells you

Accuracy is the fraction of predictions the model got right. If the model predicts "low risk" for 85 out of 100 people and 85 of them actually are low risk, accuracy is 85%. Simple.

The problem: accuracy is dominated by the majority class. If 80% of your dataset is "low risk" and 20% is "high risk," a model that predicts "low risk" for everyone achieves 80% accuracy while completely failing to identify anyone who actually needs help.

The dataset I trained on had a class imbalance problem — not as extreme as 80/20, but real. The reported 85% accuracy number includes a disproportionate amount of correctly classified low-risk cases, while the performance on the minority (high-risk) class was meaningfully worse.

The asymmetry of errors

In most classification problems, false positives and false negatives carry roughly equal cost. In mental health risk prediction, they don't.

A false positive — flagging someone as high risk when they're not — has a cost: unnecessary intervention, potential stigma, resources spent on someone who didn't need them. It's a real cost.

A false negative — failing to flag someone who is high risk — has a different cost: a person who needed support doesn't receive it. The framing in which 85% accuracy is a success doesn't distinguish between these two failure modes. A better model for this domain would explicitly optimize to minimize false negatives, accepting a higher false positive rate as the trade-off.

In technical terms: optimize for recall on the positive class, not overall accuracy. Accept lower precision. This is a deliberate, domain-specific design choice — not a default that falls out of training a model and reporting the first number you see.

XGBoost vs. Random Forest: what actually drove the decision

I used both XGBoost and Random Forest in the project. The accuracy numbers were similar. The reason to prefer one over the other in this context isn't performance — it's interpretability.

Random Forest gives you feature importances that are relatively stable and interpretable. You can say: "the three strongest predictors of high risk in this dataset were sleep quality, reported anxiety frequency, and social isolation score." That's a sentence a researcher or counselor can engage with.

XGBoost's feature importances are less stable across runs and depend more heavily on the specific boosting parameters. It tends to perform slightly better on structured tabular data, which is why it's commonly preferred for Kaggle-style competitions. But for a domain where understanding why a prediction was made matters almost as much as the prediction itself, I should have weighted interpretability more heavily and leaned toward Random Forest as the primary model.

What the Streamlit interface gets right (and wrong)

The application lets a user fill in a survey and receive a risk prediction immediately. The interface is clean and fast. That's the part that's right.

What it gets wrong: it presents the output as a binary "low risk / high risk" classification without confidence scores, without a list of which factors drove the prediction, and without any guidance on what "high risk" means or what to do about it. A risk score without context is at best useless and at worst actionable in the wrong direction.

A better interface would show confidence, show the top contributing features with their direction (e.g., "low sleep quality was the strongest factor"), and include a disclaimer that the model is a screening tool, not a diagnosis. These aren't features I built. They should be.

What this project taught me about ML on sensitive domains

Standard ML curricula teach you to maximize accuracy, minimize loss, beat a baseline. For most applications — recommendation systems, image classifiers, spam filters — those are reasonable targets. For applications that touch human health and wellbeing, the framework needs to expand.

The questions that should come before "what's the accuracy?" are: who gets harmed when the model is wrong? Which direction of error is worse? What does the user actually do with the prediction? Can they understand why it was made?

An 85% accurate model that makes its errors on the people who most need help is not a good model. It's a model that looks good until you ask the right questions.

I still list this project because the ML pipeline itself — data cleaning, feature engineering, ensemble comparison, Streamlit deployment — is solid work and I learned a lot building it. But the headline metric deserves the scrutiny I'm giving it here.