Data Engineering
What makes ML work and stay trustworthy — training and benchmark datasets, feature extraction, and honest model evaluation.
- Benchmark DatasetsData-engineering workflow
Standard, shared datasets on which different methods are compared on equal footing — like the Galaxy Zoo morphology labels or the PLAsTiCC transient-classification challenge. They let the field measure real progress rather than incomparable claims.
- Feature ExtractionData-engineering workflow
Turning raw data into the informative quantities a model uses — statistics of a light curve, cut-outs around a source, colours from photometry. Well-chosen features can make a simple model succeed; increasingly, representation learning discovers them automatically.
- Model EvaluationData-engineering workflow
Measuring honestly how well a model performs — its accuracy, completeness, and purity, whether its confidences are calibrated, and how it behaves on data unlike its training set. Careful evaluation is what separates a genuinely useful model from one that has merely memorised its examples.
- Training DatasetsData-engineering workflow
The labelled examples a model learns from. Their size, coverage, and biases largely determine how well a model works and where it fails — a classifier only knows the kinds of object it was shown, and inherits any selection effects in how those examples were gathered.