Why feature engineering is the highest-impact ML discipline
In practice, a simple model (logistic regression, decision tree) applied to well-crafted features generally beats a sophisticated model (XGBoost, neural network) applied to raw data. This observation, verified hundreds of times in Kaggle competitions and industrial projects, places feature engineering at the heart of any serious ML workflow.
The reason is fundamental: ML algorithms learn patterns in the feature space. If the features don't capture the right dimensions of the problem, even the most powerful algorithm can't extract anything relevant from them. Conversely, rich and informative features make the problem trivial for almost any model.
The data-centric AI vs model-centric AI paradigm
The ML community was long model-centric: the dataset is fixed, and you improve the algorithm and hyperparameters. Andrew Ng popularized the complementary paradigm in 2021, data-centric AI: the model is fixed (or standard), and you systematically improve the data and features instead. In his benchmarks, 10 to 40% precision improvements were achieved purely by improving features and labels, without changing the algorithm at all.
In practice, the two approaches combine. But the data-centric priority is especially relevant for industrial projects where data is heterogeneous, and where bugs more often come from poorly defined features than from a suboptimal algorithm. A regular audit of your features (distribution, completeness, coherence with the target) is more cost-effective than chasing the best hyperparameter.
Pedro Domingos, in his landmark paper 'A Few Useful Things to Know about Machine Learning', formulates this principle: 'The features used are often more important than the choice of learner.' Andrew Ng reaffirmed it in 2021 with his Data-centric AI movement, backed by empirical results showing that improving data and features systematically produced more gains than improving algorithms on real industrial projects.
Domingos, P. (2012). A Few Useful Things to Know about Machine Learning. Communications of the ACM, 55(10), 78-87.The four major variable families
Before transforming data, you need to understand its nature. Each variable type follows a different processing logic and has its own set of adapted techniques.
Numeric variables: transformations and aggregations
Numeric variables (age, income, temperature, click count) are directly usable by most algorithms, but rarely in their raw form. A highly skewed distribution (salaries, property prices, transaction amounts) can disturb linear models and distance-based algorithms. Typical transformations: logarithm to reduce skewness in positive values, square root to compress large values, Box-Cox to normalize arbitrary distributions, quantile transform to force a uniform or normal distribution.
Aggregation features by entity are often among the most predictive: for a customer scoring model, the average of the last 30 days' transactions, the standard deviation of amounts, the transaction count, the max, the sum — all computed per customer — capture historical behavior far better than individual transactions do. These groupby aggregations sit at the core of feature engineering in e-commerce, banking and telecom.
Categorical variables: encoding and cardinality management
Categorical variables (country, customer type, product category) can't be used directly by algorithms that expect numeric values. For nominal, low-cardinality variables (< 20 categories), one-hot encoding creates a binary column per category — simple and interpretable. For ordinal variables (low/medium/high, Bronze/Silver/Gold), ordinal encoding preserves the order relationship.
For high-cardinality variables (thousands of values: product ID, zip code, URL), target encoding replaces each category with the mean of the target variable — powerful but prone to data leakage. Solution: always compute target encoding inside the cross-validation folds (or use leave-one-out encoding). Weight of Evidence (WoE) is a popular alternative in credit scoring: WoE = log(% good / % bad) for each bucket, with a directly interpretable probabilistic meaning.
Temporal variables and time series
A raw date is generally not informative for a model. From a timestamp, you can extract: the day of the week (strong weekly seasonality in retail), the hour (very different behaviors morning/noon/evening/night), the month, the quarter, whether it's a holiday, the season. These cyclical features (Monday and Sunday are adjacent) can be encoded with sine/cosine functions to preserve cyclicity.
For time series, lag features (the value N periods ago: lag_1, lag_7, lag_30) and moving averages (7-day, 30-day rolling means) are frequently the most predictive. A purchase 7 days after the previous one is a far more powerful churn feature than the absolute date. Time since the last event, the count of events in the last N days, and trends (a local linear regression over the recent window) round out the standard toolkit.
Text features: TF-IDF, n-grams and embeddings
Text data (customer reviews, product descriptions, emails) needs to be transformed into numeric vectors. The classic approach: TF-IDF (Term Frequency - Inverse Document Frequency) weighs each word by its frequency in the document divided by its frequency in the corpus — words frequent in a document but rare in the corpus are informative. N-grams (bigrams, trigrams) capture word associations ('machine learning' is more informative than 'machine' and 'learning' taken separately).
The modern approach: pre-trained embeddings (Word2Vec, FastText, BERT, sentence-transformers) represent each piece of text as a dense vector of 300 to 768 dimensions capturing its semantics. A BERT embedding of a product description captures the semantic similarity between 'smartphone' and 'cell phone' that TF-IDF treats as completely unrelated. These embeddings are directly usable as features for any downstream model (XGBoost, logistic regression) — this is modern feature engineering for NLP.
Feature transformation and creation techniques
Beyond treatments specific to each variable type, cross-cutting techniques let you capture interactions, non-linearities, or domain knowledge.
Normalization and scaling: when and why
Scaling is critical for distance-based algorithms (KNN, SVM) or gradient-based ones (logistic regression, neural networks). Without normalization, a variable in euros (order of magnitude 100,000) dominates a percentage variable (order 0-1), distorting the learned weights and slowing down convergence. StandardScaler centers and scales (mean 0, standard deviation 1). MinMaxScaler compresses between 0 and 1. RobustScaler uses the median and IQR, resistant to outliers.
Tree-based ensemble methods (Random Forest, XGBoost, LightGBM) are invariant to monotonic feature transformations (normalization, log, square root) and don't need them. Applying them anyway isn't a mistake, just unnecessary. On the other hand, TF-IDF text features can benefit from L2 normalization (cosine normalization) for similarity-based models.
Interactions, ratios and polynomial features
A linear model only captures additive effects. Explicitly creating interaction features (the product of two variables, a ratio, a difference) lets you capture combinatorial effects without switching to a non-linear model. Classic examples: the 'session duration / pages viewed' ratio for engagement, 'transaction amount / 30-day average' for fraud detection, 'account age / number of transactions' for credit scoring.
scikit-learn's PolynomialFeatures systematically generates every interaction between features up to degree N — useful for exploration, but it explodes dimensionality (100 features become 5,050 features at degree 2). In practice, the relevant interactions are identified through domain knowledge or decision-tree techniques (a tree's splits naturally surface important conditions, which can potentially be exploited as interaction features).
Group-by aggregations: the groupby pattern
The most powerful feature engineering pattern in practice for transactional data: for each entity (customer, product, seller), compute aggregations over its historical transactions. For a customer: transactions_30d, total_amount_30d, avg_amount_30d, amount_std_dev, distinct_categories_count, days_since_last_purchase. For a product: views_7d, add_to_cart_rate, conversion_rate, volume-weighted average rating, return_count.
These features capture the entity's behavior and history far better than raw variables do. The difficulty is point-in-time correctness: at training time, you must only use transactions that occurred before the example's date, never future data. A feature store with temporal support solves this systematically (see the Feature Stores section).
Target encoding is prone to data leakage if computed over the entire dataset before the train/test split: target means computed on test examples 'contaminate' the training set. Always compute target encoding inside the cross-validation folds (scikit-learn 1.3+'s TargetEncoder does this automatically), or use smoothed target encoding (a weighted average with the global mean, based on category size) to reduce overfitting on small categories.
Feature selection: fewer features, better model
Adding redundant or uninformative features doesn't improve performance: it increases noise, slows training, and can hurt generalization (the curse of dimensionality). Feature selection identifies the optimal subset of variables.
Statistical filters: fast screening
Filters evaluate each feature independently of the algorithm — maximum speed, scalable to millions of features. Pearson correlation measures the linear relationship with a numeric target. The chi-square test evaluates independence between a categorical variable and the target. Mutual information measures any form of dependence, linear or not, based on information theory.
These methods are fast but ignore interactions between features: a variable can be non-informative on its own but highly predictive combined with another. They serve as a pre-filtering step to drop clearly useless features (near-zero variance, zero correlation with the target) before applying costlier methods.
Wrapper methods: RFE and sequential selection
Wrapper methods evaluate subsets of features by training the model on each subset. RFE (Recursive Feature Elimination) trains the model, drops the least important feature according to the model's weights, and repeats until reaching the target number of features. scikit-learn's RFECV (RFE with Cross-Validation) automatically picks the optimal number of features via cross-validation.
Sequential selection (forward: add one feature at a time; backward: remove one at a time) explores the space more systematically, but is computationally expensive (O(n^2) iterations). scikit-learn's SequentialFeatureSelector implements both. These methods fit when the number of candidate features is moderate (< 500) and you can afford several hours of compute.
Embedded methods: Lasso, tree importances and SHAP
Embedded methods select features during model training. L1 regularization (Lasso) pushes unimportant coefficients exactly to zero, automatically selecting the relevant features — very effective for linear models. scikit-learn's SelectFromModel lets you use any estimator with a feature_importances_ or coef_ attribute to select the most important features.
SHAP (SHapley Additive exPlanations) computes each feature's marginal contribution to each individual prediction, then aggregates them into a global importance score. Unlike Random Forest importances (biased toward high-cardinality and strongly correlated variables), SHAP is theoretically grounded (Shapley values from cooperative game theory) and works with any model, including neural networks and fine-tuned LLMs.
Andrew Ng (founder of Google Brain, deeplearning.ai) published the 'data-centric AI' concept in 2021, in contrast with traditional 'model-centric AI'. In his benchmarks on industrial projects, systematically improving features and labels — without changing the algorithm — produced 10 to 40% precision gains. His MLOps program spends more time on data quality and feature engineering than on model tuning.
Ng, A. (2021). A Chat with Andrew on MLOps: From Model-centric to Data-centric AI. DeepLearning.AI.Feature engineering by domain: e-commerce, fintech and NLP
Domain knowledge is the most powerful source of relevant features. Data scientists who understand the domain extract features that automation doesn't detect. Here are the characteristic patterns for the three most common domains.
E-commerce and recommendation
The most predictive features in e-commerce: recency (days since the last purchase), frequency (number of orders over 90 days), monetary value (average basket, total sum) — the RFM (Recency-Frequency-Monetary) model forms a solid feature foundation for churn and customer scoring. Browsing behavior features: session duration, number of pages viewed, bounce rate, pages visited in the 7 days before the order.
Product features: product page conversion rate, expected vs actual delivery time, return rate, volume-weighted average rating, position in search results. Contextual features: device (mobile vs desktop), time of order, day of week, source marketing campaign. E-commerce feature engineering combines temporal aggregations across multiple granularities (7d, 30d, 90d, 1 year).
Fintech and credit scoring
Credit scoring is the domain where feature engineering is most formalized. Classic scoring features: debt-to-income ratio (DTI), credit utilization rate (balance / limit), credit history length, credit type mix (revolving, installment, mortgage), number of recent inquiries. Each is bucketed into categories rather than left as a continuous value, to capture threshold effects (a DTI of 35% is categorically different from 36%).
Weight of Evidence (WoE) and Information Value (IV) are the reference tools of bank scoring: WoE = ln(good distribution / bad distribution) for each bucket of each variable, IV = sum of WoE x (good distribution - bad distribution). An IV < 0.02: useless feature. 0.02-0.1: weak predictor. 0.1-0.3: moderate predictor. > 0.3: strong predictor. WoE encoding turns every variable into a continuous value interpretable in terms of relative risk.
NLP and text classification
For text classification tasks (sentiment analysis, categorization, spam detection), the feature engineering pipeline follows an increasing progression of complexity. Level 1 — statistical features: text length, word count, sentence count, uppercase/lowercase ratio, presence of specific punctuation (!, ?, ...). These simple features are often very predictive for spam and extreme reviews.
Level 2 — TF-IDF with n-grams: tokenization plus stop-word removal plus stemming/lemmatization plus TF-IDF on unigrams and bigrams. A standard scikit-learn pipeline using TfidfVectorizer. Level 3 — pre-trained embeddings: sentence-transformers (all-MiniLM-L6-v2, paraphrase-multilingual-MiniLM) produce 384-dimension vectors encoding semantics. These vectors are directly usable as features for XGBoost or as a classification head on a fine-tuned BERT model. Embeddings capture the semantics that TF-IDF misses.
Feature stores: industrializing feature engineering
In mature ML organizations, ad-hoc manual feature engineering creates problems: duplicated computation across teams, inconsistency between training and inference (training-serving skew), and the impossibility of reusing features from one project to another. A feature store solves these problems by centralizing feature definition, computation, storage and serving.
Offline / online architecture
The offline store holds batch features in a data warehouse (BigQuery, Snowflake, Redshift) or a data lake (Delta Lake, Iceberg). It's used for model training and building historical datasets. Computations are scheduled (hourly, daily, depending on the freshness required) and features are versioned. The online store holds the same features at low latency (Redis, DynamoDB, Cassandra, Bigtable) to serve them in real time at prediction time — target latency under 10ms.
A central catalog ensures offline and online features stay synchronized, and that every feature is documented (business definition, owner, last update date, applied transformations, version of the computation code). When a Data Scientist trains a model, they pull features from the offline store using the exact same computation logic used in production — eliminating a major source of silent bugs.
Point-in-time correctness: the invariant of every feature store
Point-in-time correctness guarantees that, when building a training dataset, the features attached to each example (customer, transaction) match the values that were available at the time of the event — not current values, not tomorrow's values. Without this guarantee, you end up computing aggregations over future data: did the customer make 5 transactions at the time of purchase, or are you using their current count of 47?
Feature stores implement this guarantee through point-in-time temporal queries: 'give me entity X's features as they existed on date T'. Feast supports point-in-time joins natively via its get_historical_features function. Without a feature store, reproducing this guarantee manually with temporal SQL joins (WHERE event_timestamp <= label_timestamp) is possible but error-prone and hard to audit systematically.
Feast (open-source, cloud-agnostic, integrates with Redis, BigQuery, Snowflake) for teams that want control and portability. Tecton (enterprise SaaS, formerly Uber Michelangelo, AWS and GCP) for large organizations. Databricks Feature Engineering (native integration with Delta Lake, MLflow and Unity Catalog) for teams already on Databricks. Vertex AI Feature Store (GCP managed, no ops) and SageMaker Feature Store (AWS) for cloud-native teams. The main criterion: adoption friction — the best feature store is the one teams actually use.
Memorizing feature engineering with spaced repetition
Feature engineering relies on a mix of statistical concepts (WoE, TF-IDF, mutual information), algorithmic reflexes (when to normalize, when to use target encoding, how to handle point-in-time correctness) and domain knowledge (RFM features for retail, DTI for credit, embeddings for NLP). The sheer number of techniques and their specificity make passive memorization inefficient.
memia offers flashcard decks covering feature engineering techniques, feature selection and production patterns (feature stores). Every card is AI-generated and validated, with concrete examples and mnemonics. By anchoring these concepts through FSRS spaced repetition, they become reflexes you can immediately apply to your projects.
Recurring themes: (1) One-hot encoding vs target encoding vs WoE, and when to use each. (2) Why normalize for KNN and SVM but not for XGBoost. (3) What data leakage is and how to detect it in target encoding. (4) How to create per-entity aggregation features (the groupby pattern). (5) SHAP vs Random Forest feature importances. (6) What point-in-time correctness is and why a feature store needs it.
Explore the Data & AI cluster
Frequently asked questions about feature engineering
What exactly is feature engineering?
Feature engineering is the process of transforming raw data into variables (features) usable by a machine learning algorithm. It includes cleaning, creating new variables (aggregations, interactions, temporal extractions), encoding categorical variables, normalization, and selecting the most informative features. It's often the step that determines 80% of a model's performance.
Why is feature engineering more important than algorithm choice?
ML algorithms learn patterns in the feature space. Poorly built features mean no useful pattern to learn, regardless of the algorithm. Rich features mean any algorithm works. Pedro Domingos (2012) and Andrew Ng (2021) have both documented this principle: improving features systematically produces more gains than optimizing algorithms on industrial projects.
What is the difference between feature engineering and feature selection?
Feature engineering creates new variables from raw data (transformation, combination, extraction, aggregation). Feature selection chooses among existing variables the ones that are most informative. You first create as many relevant features as possible, then select those that contribute the most — the two are complementary.
Do you always need to normalize your features?
No — it depends on the algorithm. Normalization needed: KNN, SVM, logistic regression, neural networks (distance-based or gradient-based). Normalization unnecessary: Random Forest, XGBoost, LightGBM (trees, invariant to monotonic transformations). The rule of thumb: normalize for every algorithm except tree-based ensemble methods.
What is target encoding and when should you use it?
Target encoding replaces each category of a categorical variable with the mean of the target variable. Ideal for high-cardinality variables (hundreds or thousands of categories) where one-hot encoding would explode dimensionality. Risk: data leakage if computed over the whole dataset. Solution: always compute it inside the cross-validation folds. scikit-learn 1.3+ includes a TargetEncoder that does this automatically.
What is Weight of Evidence (WoE)?
WoE is an encoding technique for credit scoring: WoE = ln(distribution of good cases / distribution of bad cases) for each bucket of a variable. It turns every variable into a continuous value interpretable in terms of relative risk. Information Value (IV) measures the variable's overall predictive power: IV < 0.02 = useless, 0.1-0.3 = moderate predictor, > 0.3 = strong predictor.
What is a feature store and when do you need one?
A feature store centralizes the definition, computation, storage and serving of ML features — an offline store (batch, for training) and an online store (real time, for inference). It becomes necessary when several teams share features, when you observe inconsistencies between training and production (training-serving skew), or when point-in-time correctness is critical. Tools: Feast, Tecton, Databricks Feature Engineering, Vertex AI Feature Store.
How does SHAP help with feature selection?
SHAP computes each feature's contribution to each individual prediction (game theory — Shapley values), then aggregates them into a global importance score. Advantage over Random Forest feature importances: SHAP isn't biased toward high-cardinality variables, works for any model (including black boxes), and produces local explanations (why this specific prediction) on top of global importances.
What is point-in-time correctness?
Point-in-time correctness guarantees that, when building a training dataset, the features attached to each example match the values available at the time of the event — not current values. Without this guarantee, you create temporal data leakage: aggregations include future data. A feature store with temporal queries ('X's features at time T') solves this systematically.
When should you use TF-IDF vs embeddings for text features?
TF-IDF (with n-grams): fast, interpretable, works well for short texts with a stable vocabulary, spam classification, language detection. Embeddings (sentence-transformers, BERT): capture semantics (synonyms, context), better performance on similarity and comprehension tasks, robust to vocabulary variation. In practice: TF-IDF as a fast baseline, embeddings for maximum performance or when vocabulary varies.