2025 · Data Science
Big Data Predictive Analysis of Drinking-Water Quality
PySpark pipeline over 1.2M+ rows predicting drinking-water safety against Indonesian health-ministry standards.
Private build
Problem
Predicting drinking-water quality in Indonesia against the Ministry of Health (Kemenkes RI) indicators was not available automatically, and the dataset was far too large to process by hand.
Role & constraints
Data-science work in a team of two, from June to July 2025, as a course project. The code is private.
Approach
I built an end-to-end PySpark pipeline over more than 1.2 million rows: cleaning, handling anomalous values, imputing nulls, and pivoting the features into a model-ready shape. A logistic-regression classifier in Spark MLlib predicts whether a water sample is safe to drink, evaluated with accuracy and AUC.
Result
Accuracy of 88.62 percent and an AUC of 0.86, with the whole pipeline reproducible instead of a one-off notebook.
Lessons
Cleaning data at this scale, across many features, is where most of the work and most of the risk lives.