AI, Machine Learning, Deep Learning: the key distinctions
Artificial intelligence (AI) is the general field aimed at creating systems capable of performing tasks that normally require human intelligence. Machine learning (ML) is a subset of it: instead of explicitly programming rules, you feed data to the system so it can infer the patterns itself. Deep learning is in turn a subset of ML, built on multi-layer artificial neural networks.
This distinction is not purely academic. A traditional ML system like logistic regression is interpretable, fast to train and deployable on standard hardware. A deep learning model like a transformer requires GPUs, millions of data points and a significant compute budget. Choosing the right approach depends on the problem, how much data is available, and operational constraints.
Learning vs rule-based programming
The fundamental difference between ML and classic programming is the direction of the derivation. In classic programming: data + rules → results. In ML: data + expected results → rules (the model). This inversion is powerful for problems where the rules are too complex or too numerous to code by hand — speech recognition, fraud detection, content recommendation.
For a spam detection system, hand-coding rules (word lists, blocked senders) quickly hits its limits. An ML model automatically learns hundreds of discriminating features from examples of spam and legitimate emails, generalizes to new patterns, and adapts to new spam techniques through retraining.
Symbolic AI vs connectionist AI
Symbolic AI (expert systems, formal logic, Bayesian networks) represents knowledge explicitly as rules and facts. It is interpretable and robust on well-defined domains, but fragile in the face of uncertainty and hard to scale. Connectionist AI (neural networks, deep learning) learns implicit representations from data. It is more flexible and higher-performing on complex perceptual tasks, but less interpretable.
The recent trend is toward hybrid approaches: a connectionist LLM can generate code or rules (symbolic), while a symbolic reasoning system can constrain a neural model's outputs. The best AI systems in 2025 — such as neuro-symbolic architectures for planning — draw on both approaches.
Alan Turing laid the theoretical foundations of AI in 1950 with his paper 'Computing Machinery and Intelligence' (which introduced the Turing test). Arthur Samuel coined the term 'machine learning' in 1959 while working on a checkers program capable of improving through experience. Frank Rosenblatt invented the perceptron in 1958 — the first trainable artificial neuron model — laying the groundwork for modern deep learning.
Samuel, A.L. (1959). Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development.The major learning paradigms
ML splits according to the nature of the available learning signal. Each paradigm addresses distinct types of problems and calls for different algorithms.
Supervised learning: learning with labels
Supervised learning uses (input, expected output) pairs to learn a mapping function. There are two cases: classification (a discrete output — spam/not-spam, diagnosis, product category) and regression (a continuous output — a property's price, a forecast temperature, a risk score). Label quality is critical: a poorly annotated dataset produces a biased model, regardless of which algorithm you choose.
Typical examples: bank fraud detection (binary classification), credit scoring (regression or classification), image recognition (multi-class classification), churn prediction (binary classification), energy consumption forecasting (regression). This is the most common paradigm: about 85% of ML models in production are supervised, because most business systems naturally accumulate labeled historical data (approved/rejected transactions, resolved/escalated tickets).
Unsupervised learning: discovering structure
When labels are absent or too costly to obtain, unsupervised learning extracts latent structure from the data alone. The main tasks are clustering (K-Means, DBSCAN, hierarchical — grouping similar individuals), dimensionality reduction (PCA reduces explained variance; UMAP and t-SNE preserve neighborhood relationships for visualization; autoencoders learn a compressed representation) and anomaly detection (Isolation Forest, One-Class SVM — identifying atypical points without positive labels).
Concrete applications: customer segmentation for personalized marketing, network intrusion detection without a catalog of known attacks, content recommendation by similarity (collaborative filtering), image compression. Evaluation is harder than in supervised learning because there is no reference 'correct answer' — you rely on internal metrics (silhouette score, inertia for K-Means) or business validation.
Reinforcement learning: learning through experience
An agent interacts with an environment, receives rewards or penalties, and learns a policy that maximizes cumulative long-term reward. The central challenge is the exploration/exploitation trade-off: should the agent pick the action known to be good (exploitation) or explore a new, potentially better action (exploration)?
This paradigm has produced the most spectacular systems: AlphaGo (DeepMind, 2016) defeating the world Go champion, Atari-playing AIs outperforming humans across dozens of games, high-frequency trading algorithms optimizing complex strategies. RLHF (Reinforcement Learning from Human Feedback) is at the heart of aligning large language models — it's the technique used to tune GPT-4, Claude or Gemini's responses to be helpful and harmless.
Semi-supervised and self-supervised learning: the future of learning
Semi-supervised learning uses a small labeled dataset alongside a large unlabeled one — highly relevant when labeling is expensive (medical annotations, legal analysis). A model pre-trained on the unlabeled data is then fine-tuned on the labeled examples.
Self-supervised learning generates its own labels from the data: BERT pre-trains by predicting masked tokens, contrastive vision models (SimCLR, MoCo) learn by comparing augmented versions of the same image. This approach produces general representations (embeddings) transferable to many downstream tasks with little labeled data — it is the dominant paradigm in modern deep learning.
Supervised learning accounts for roughly 85% of ML use cases in production, according to Kaggle and O'Reilly ML surveys. The main reason: the availability of labeled historical data in business systems (CRM, ERP, ticketing tools). Unsupervised learning dominates exploration and segmentation. Reinforcement learning remains mostly confined to research labs and a few specific domains (robotics, trading, games).
The fundamental algorithms to know
The ML algorithmic landscape is vast, but a handful of families cover the overwhelming majority of use cases. Knowing them lets you pick a good starting point and interpret results.
Linear and logistic regression: the mandatory starting point
Linear regression predicts a continuous value (y = wX + b) by minimizing mean squared error. Simple, interpretable and fast, it makes an excellent baseline for any regression problem. L1 regularization (Lasso — sets some coefficients to zero, doing automatic feature selection) and L2 (Ridge — penalizes large coefficients) control overfitting.
Logistic regression extends this principle to binary classification by applying a sigmoid function to produce a probability between 0 and 1. It is widely used in regulated domains (credit, healthcare, justice) for its interpretability: the coefficients directly indicate the impact of each variable on the event's probability, and the outputs are calibrated into usable probabilities.
Decision trees, Random Forest and XGBoost: the king of tabular data
A decision tree learns hierarchical segmentation rules (if age > 35 AND income > 50k THEN low_risk). Intuitive and interpretable, but very prone to overfitting without depth constraints. Ensemble methods fix this.
Random Forest builds hundreds of trees on random subsamples (bagging plus random feature subsets) and averages their predictions. XGBoost and LightGBM use gradient boosting: each tree corrects the residual errors of the previous model by minimizing the cost function via gradient descent. XGBoost has dominated Kaggle competitions on tabular data since 2015 — it combines high performance, built-in L1/L2 regularization, native handling of missing values, and GPU support. LightGBM is Microsoft's faster alternative for large datasets.
SVM and KNN: decision boundaries and neighborhoods
SVM (Support Vector Machine) finds the decision boundary that maximizes the margin between classes — relying only on the support vectors (the examples closest to the boundary). The kernel trick implicitly projects the data into a higher-dimensional space where it becomes linearly separable: the RBF kernel for circular boundaries, polynomial kernels, and so on. SVM is effective on small datasets with many features (text classification, bioinformatics).
KNN (K-Nearest Neighbors) is the most intuitive algorithm: to predict a new example's class, you find its k nearest neighbors in the training data and vote. There is no explicit training step — the prediction happens at query time. Downsides: slow at prediction time on large datasets (O(n) per query), sensitive to unnormalized features and irrelevant data. Useful as an interpretable baseline and for recommendation systems.
Neural networks and deep learning: power on unstructured data
A neural network is made of layers of artificial neurons connected by adjustable weights. Learning happens through backpropagation: the output error is computed and the weights are adjusted backward through the network via the chain rule. Nonlinear activation functions (ReLU, Sigmoid, GELU) let the network learn complex relationships.
For unstructured data, specialized architectures consistently outperform traditional algorithms: CNNs (Convolutional Neural Networks) for computer vision, transformers for natural language processing (BERT, GPT, Llama), GNNs (Graph Neural Networks) for graph-structured data. Deep learning needs more data, compute (GPU/TPU) and expertise than classic algorithms — but transfer learning and pre-trained foundation models have dramatically cut the amount of data needed for specific tasks.
The end-to-end ML pipeline
An ML model does not live in isolation. From data collection to production deployment, a structured ML pipeline is essential for producing reliable, reproducible models.
Data preparation and cleaning
Data scientists spend 60 to 80% of their time on data, on average. Typical operations include: imputing missing values (median or mean for continuous variables, mode for categorical ones, or a predictive model for complex cases), normalizing and standardizing distributions (StandardScaler, MinMaxScaler), detecting and handling outliers (IQR, z-score, Isolation Forest), and encoding categorical variables (one-hot encoding for few categories, target encoding for high-cardinality categories, ordinal encoding for ordered categories).
Handling class imbalance is critical for fraud detection, rare disease detection and other problems where the positive class is rare (< 5%). Solutions: oversampling the minority class (SMOTE — Synthetic Minority Oversampling Technique, which generates synthetic examples), undersampling the majority class, adjusting class_weight in the model (telling it to penalize errors on the minority class more heavily).
Feature engineering: the primary source of value
Feature engineering transforms raw variables into more informative representations for the model. Examples: from a transaction date, extract the hour, day of week, month, and whether it's a public holiday — each potentially predictive for fraud detection. From a text field, extract TF-IDF, BERT embeddings, length, uppercase/lowercase ratio.
This is often the step that produces the biggest performance gain, far more than picking a fancier algorithm. Feature selection techniques (RFECV — Recursive Feature Elimination with Cross-Validation, SHAP importance, mutual information) let you drop useless or harmful features that hurt generalization.
Training and hyperparameter optimization
Training means adjusting the model's parameters (a neural network's weights, a regression's coefficients) on the training data by minimizing a cost function. Hyperparameters (learning rate, max tree depth, number of layers, dropout rate) control the learning process itself — they are not learned by the model but chosen by the data scientist.
Hyperparameter optimization is done on the validation set: grid search (exhaustive exploration of a value grid), random search (random sampling — more efficient than grid search for an equal budget), Bayesian optimization with Optuna or Hyperopt (builds a probabilistic model of performance as a function of hyperparameters and intelligently explores the space). The train/validation/test split is non-negotiable: the test set must never be seen during training or tuning.
Interpretability and explainability: SHAP and LIME
Interpretability is the ability to understand why a model makes a specific prediction. It is critical in regulated domains (credit, insurance, justice — GDPR requires explaining automated decisions) and for debugging model bias.
SHAP (SHapley Additive exPlanations, Lundberg & Lee 2017) is the dominant method: it quantifies each feature's contribution to a specific prediction based on game theory (Shapley values). SHAP works for any model (black-box or not) and produces both local explanations (for a specific prediction) and global ones (feature importance across the whole dataset). LIME (Local Interpretable Model-agnostic Explanations) is an alternative that locally approximates the model's behavior with a simple, interpretable model.
Overfitting happens when a model memorizes noise from the training set instead of learning generalizable patterns. Symptom: excellent performance on the train set (99% accuracy), mediocre on the test set (72% accuracy). Remedies, in order of effectiveness: (1) More training data. (2) Simplify the model (fewer layers, smaller max tree depth). (3) L1/L2 regularization or dropout. (4) Data augmentation. (5) Cross-validation to catch overfitting earlier.
How to evaluate an ML model
Choosing the right evaluation metrics is as important as choosing the algorithm. A poorly chosen metric leads to shipping a model that looks good on paper but fails at the real business problem.
Classification metrics: beyond accuracy
Accuracy is misleading on imbalanced datasets: a model that always predicts 'negative' reaches 99% accuracy if only 1% of cases are positive — the model is useless but its stats look good. Complementary metrics: precision (among predicted positives, how many are truly positive — minimizes false positives), recall (among true positives, how many are detected — minimizes false negatives, critical in fraud or disease detection), F1-score (the harmonic mean of precision and recall, balancing both).
The ROC (Receiver Operating Characteristic) curve plots the true positive rate against the false positive rate across every decision threshold. AUC (Area Under the Curve) summarizes it as a single number: 0.5 is random, 1.0 is perfect. The Precision-Recall curve and its AUC (AP — Average Precision) are preferable for heavily imbalanced problems, since they are less sensitive to a dominant true-negative count.
Regression metrics: MAE, RMSE, R²
For regression, MAE (Mean Absolute Error) measures the average absolute error — easy to interpret (in euros, in degrees), robust to outliers. MSE (Mean Squared Error) penalizes large errors more heavily (useful when outliers are costly — for instance, a 100-euro error is worse than 10 times a 10-euro error). RMSE (Root MSE) is the square root of MSE, in the same unit as the target variable — often preferred for its readability.
R² (the coefficient of determination, between 0 and 1) measures the proportion of variance explained by the model compared to a model that always predicts the mean. An R² of 0.85 means the model explains 85% of the data's variability, with the remaining 15% being noise or factors not included. R² can be negative if the model is worse than predicting the mean.
Cross-validation and evaluation robustness
K-fold cross-validation splits the data into k partitions, trains k models (each excluding one partition) and averages the scores. It gives a more robust estimate of real-world performance than a simple train/test split, especially on moderately sized datasets. Typical values of k: 5 or 10. Stratified k-fold preserves the class proportions in each fold — essential for imbalanced datasets.
For time series, cross-validation must strictly respect chronological order (TimeSeriesSplit in scikit-learn, or Walk-Forward Validation): the test fold must always come after the training fold. Mixing data from different periods without respecting temporal order creates look-ahead bias — the model sees the future during training, producing artificially good metrics.
The confusion matrix breaks predictions into 4 categories: true positives (TP), true negatives (TN), false positives (FP — false alarms) and false negatives (FN — missed cases). Precision = TP/(TP+FP). Recall = TP/(TP+FN). Depending on the business context, errors don't cost the same: for cancer detection, a false negative (a missed disease) is far more serious than a false positive (a false alarm followed by an extra exam) — so you optimize recall at the expense of precision.
Tools, the Python ecosystem, and when to use ML
Machine learning in practice runs on a mature Python ecosystem, and a choice between open source and managed cloud platforms.
The Python ML ecosystem
scikit-learn is the reference library for classic ML (regression, classification, clustering, preprocessing, metrics) — a uniform API, excellent documentation, easy to integrate. XGBoost and LightGBM are the dominant libraries for gradient boosting on tabular data. PyTorch (Meta) and TensorFlow/Keras (Google) are the deep learning frameworks — PyTorch dominates research (eager execution, debuggability), while TensorFlow/Keras historically led in production (though PyTorch has gained ground in 2023-2025).
Hugging Face Transformers has become indispensable for NLP: access to thousands of pre-trained models (BERT, GPT-2, LLaMA, Mistral) through a unified API for fine-tuning and inference. Pandas and NumPy for data manipulation. Matplotlib, Seaborn and Plotly for visualization. MLflow for experiment tracking and the model registry. Most production ML projects combine scikit-learn (preprocessing), XGBoost or LightGBM (model) and MLflow (tracking).
When to use ML vs a rule-based system
ML isn't always the right answer. It's justified when: (1) the problem is too complex for manual rules (speech recognition needs thousands of phonological rules that are impossible to maintain by hand), (2) patterns change over time (fraud behaviors constantly shift — a model can be retrained), (3) abundant, representative labeled historical data is available.
A rule-based system is preferable when: the rules are stable, few, and explicitly known (form validation, format checks), full interpretability is required by regulation (judicial decisions), labeled data is insufficient, or latency is critical and compute resources are limited. The rule of thumb: always start with a simple rule-based system, measure its limits, then introduce ML for the cases where the rules fail.
Since its publication by Chen & Guestrin (KDD 2016), XGBoost has established itself as the dominant algorithm on structured tabular data. Its strength comes from combining: gradient boosting with L1/L2 regularization (avoids overfitting), native handling of missing values, GPU support, and algorithmic optimizations (approximate tree learning, cache-aware access). On Kaggle, the majority of wins on tabular data between 2015 and 2022 used XGBoost or LightGBM. The rise of transformers for tabular data (TabNet, FT-Transformer) has not yet overturned this dominance in production.
Chen, T. & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. KDD 2016.Memorizing machine learning with spaced repetition
Machine learning is a dense field: technical vocabulary (overfitting, gradient boosting, AUC-ROC, SHAP), fine-grained algorithmic distinctions (Random Forest vs XGBoost, precision vs recall, MAE vs RMSE), and classic pitfalls (data leakage, look-ahead bias, class imbalance). Spaced repetition is the most scientifically effective method for anchoring these concepts for the long term.
memia offers flashcard decks covering ML fundamentals, algorithms, model evaluation, interpretability and the production pipeline. Every card is AI-generated and validated, with hints and mnemonics to speed up memorization. In 10 minutes a day, the key concepts become reflexes — essential before an ML Engineer or Data Scientist interview.
Recurring questions in Data Scientist and ML Engineer interviews: (1) Precision vs recall, and which one to optimize depending on context. (2) How to detect and fix overfitting. (3) Why XGBoost is robust on tabular data. (4) What data leakage is and how to avoid it. (5) When to use AUC-ROC vs a PR curve. (6) What SHAP is and how to interpret it. (7) Supervised vs unsupervised learning, with concrete examples.
Explore the Data & AI cluster
Frequently asked questions about machine learning
What is the difference between machine learning and artificial intelligence?
Artificial intelligence is the general field aimed at creating intelligent systems. Machine learning is a subset that learns patterns from data rather than following explicitly programmed rules. All ML is AI, but not all AI is ML — expert systems, formal planning and symbolic logic are AI without being ML.
What is the difference between machine learning and deep learning?
Deep learning is a subset of machine learning that uses artificial neural networks with multiple hidden layers. It excels on unstructured data (images, text, audio) with large volumes of data, but requires GPUs, expertise and a significant budget. Classic ML (regression, XGBoost, SVM) often remains superior on structured tabular data and is more interpretable and less resource-hungry.
Do you need to know how to code to do machine learning?
Python is the dominant language, with scikit-learn, PyTorch and XGBoost. Some knowledge of statistics and linear algebra is useful. No-code tools like Google AutoML, DataRobot or H2O AutoML let you build models without code, but understanding the fundamentals remains necessary to interpret results and avoid pitfalls (data leakage, overfitting).
How much data do you need to train an ML model?
It depends on the complexity of the problem. Logistic regression works with a few hundred examples; XGBoost needs a few thousand to perform well; a deep neural network requires millions. Transfer learning (starting from a pre-trained model) dramatically reduces this need. Quality beats quantity: 10,000 well-labeled examples are worth more than 1 million noisy data points.
What is overfitting and how do you avoid it?
Overfitting happens when a model memorizes noise from the training set instead of learning generalizable patterns. Symptom: excellent train performance (99% accuracy) but mediocre test performance (72% accuracy). Remedies: (1) More data. (2) Simplify the model. (3) L1/L2 regularization or dropout. (4) Cross-validation to catch overfitting early.
What is the difference between precision and recall?
Precision = among positive predictions, how many are correct? (minimizes false positives — legitimate email wrongly sent to trash). Recall = among true positive cases, how many are detected? (minimizes false negatives — undetected cancer). The F1-score is their harmonic mean. The choice depends on the business cost of errors: for detecting a serious disease, a false negative is far more costly than a false positive — so you optimize recall.
Which ML algorithm should I choose for my problem?
For tabular data: start with logistic/linear regression (an interpretable baseline), then XGBoost or LightGBM (performance). For images: CNNs or vision transformers. For text: BERT/RoBERTa transformers or fine-tuning an LLM. The general rule: favor simplicity and interpretability when business constraints require it (credit, healthcare, justice), raw performance otherwise.
What is deep learning and when should you use it?
Deep learning uses neural networks with multiple hidden layers. It excels on unstructured data (images, sound, text) and with large volumes of data. It is less suited to classic tabular data (where XGBoost often dominates), requires more resources (GPU), and is less interpretable. Transfer learning (starting from a pre-trained foundation model) has made deep learning accessible with less data.
What is SHAP and why does it matter?
SHAP (SHapley Additive exPlanations) quantifies each feature's contribution to a specific prediction based on game theory (Shapley values). It works for any model (black-box or not) and produces both local explanations (why this prediction for this individual) and global ones (which features matter most overall). SHAP is essential in regulated domains and for debugging model bias.
How do I evaluate whether my model is good?
Choose metrics aligned with the business objective: F1-score or AUC-ROC for classification on imbalanced classes, RMSE or MAE for regression. Compare against a simple baseline (predicting the majority class or the mean). Test on truly unseen data. Evaluate performance by segment (demographic groups, time periods) to catch localized bias or degradation.