Lesson 5: Data Preprocessing & Feature Engineering

Data quality determines model quality. This lesson covers essential techniques for preparing raw data for machine learning, including handling missing values, scaling features, encoding categorical variables, and engineering new features to improve model performance.

Why Data Preprocessing Matters

"Garbage in, garbage out." Machine learning models are only as good as the data they're trained on. Real-world data is often messy, incomplete, and inconsistent. Data preprocessing transforms raw data into a clean, structured format suitable for modeling.

Studies show that data scientists spend approximately 80% of their time on data cleaning and preparation. Mastering these skills dramatically improves model performance and training efficiency.

Handling Missing Values

Missing data is common in real datasets. There are several strategies to handle it:

// Handling missing values with ML.NET
var pipeline = mlContext.Transforms
    // Replace missing numeric values with mean
    .ReplaceMissingValues("Age", replacementMode: MissingValueReplacingEstimator.ReplacementMode.Mean)
    // Replace missing text values with mode (most frequent)
    .ReplaceMissingValues("Category", replacementMode: MissingValueReplacingEstimator.ReplacementMode.Mode)
    .Append(mlContext.Transforms.Text.FeaturizeText("Features", "Text"));

Feature Scaling and Normalization

Machine learning algorithms are sensitive to feature magnitude. Features with larger scales can dominate the learning process. Feature scaling brings all features to a similar range.

• Normalization (Min-Max): Scales values to [0, 1] range. Formula: (x - min) / (max - min)
• Standardization (Z-score): Centers data around 0 with standard deviation 1. Formula: (x - mean) / std_dev
• Log Scaling: For skewed distributions with outliers

// Feature scaling in ML.NET
var pipeline = mlContext.Transforms
    // Normalize features to [0,1] range
    .NormalizeMinMax("Features")
    // OR standardize (zero-mean, unit variance)
    .Append(mlContext.Transforms.NormalizeLogMeanVariance("Features"))
    .Append(trainer);

Encoding Categorical Variables

Machine learning algorithms work with numerical inputs. Categorical variables (like color, city, or category) must be converted to numbers.

// Encoding categorical data
var pipeline = mlContext.Transforms
    // Convert string categories to numbers (ordinal encoding)
    .Conversion.MapValueToKey("Color")
    // OR one-hot encode (creates multiple binary columns)
    .Append(mlContext.Transforms.OneHotEncoding("Color"))
    .Append(trainer);

Feature Engineering

Feature engineering is the art of creating new features from existing data that better represent underlying patterns. Good features can dramatically improve model performance.

Examples of feature engineering:

• Polynomial Features: Create x², x³, or interaction terms (x × y)
• Date Features: Extract day, month, year, day-of-week from timestamps
• Text Features: Word count, sentiment score, character length
• Domain Features: Calculate ratios, differences, or derived metrics

// Feature engineering examples
var pipeline = mlContext.Transforms
    // Create polynomial features
    .CopyColumns("Features_squared", "Features")
    .Append(mlContext.Transforms.Math.MultiplyBy("Features_squared", 
        estimatorChain: mlContext.Transforms.Concatenate("Features_squared", "Features")))
    // Extract parts from text
    .Append(mlContext.Transforms.Text.FeaturizeText("TextFeatures", "Description"))
    // Normalize
    .Append(mlContext.Transforms.NormalizeMinMax("Features"));

Lesson Summary