Machine learning interview questions

The same data, with and without labels

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.linear_model import LogisticRegression

X, y = make_blobs(n_samples=300, centers=3, random_state=42)

clf = LogisticRegression().fit(X, y)              # supervised: learns from X and the labels y
print(clf.predict(X[:8]), y[:8])                  # [2 2 0 1 2 1 0 1] [2 2 0 1 2 1 0 1]

km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)   # unsupervised: X only
print(km.labels_[:8])                             # [1 1 0 2 1 2 0 2]

k-means found the same three groups, but numbered them differently: without labels, it has no way to know what each group should be called. Naming and interpreting clusters is up to you.

Other kinds you may be asked about

• Semi-supervised: a few labelled examples plus many unlabelled ones.
• Self-supervised: the labels come from the data itself. Large language models are pre-trained this way, by predicting the next token of ordinary text.
• Reinforcement learning: an agent learns from rewards for its actions, as in game-playing or robotics.

Follow-ups

• Classification or regression: predicting a customer's age? Regression, because age is a number. Predicting an age band such as "18–25" would be classification.
• Can unsupervised learning help a supervised model? Yes. Cluster labels or PCA components can become features, and anomaly scores can flag rows to check before training.

Seeing it in the scores

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=600, n_features=20, n_informative=5, flip_y=0.1, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

for depth in (1, 4, None):
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"max_depth={depth}: train {tree.score(X_train, y_train):.2f}, test {tree.score(X_test, y_test):.2f}")

max_depth=1: train 0.81, test 0.77
max_depth=4: train 0.87, test 0.79
max_depth=None: train 1.00, test 0.73

A one-split tree is too simple. An unlimited tree memorises the training set, 10% deliberately mislabelled rows included, and scores worst on new data. The depth-4 tree generalises best.

How to fix each

The trade-off in one line: a model's expected error on new data is bias² + variance + irreducible noise. You can trade bias against variance, but not remove the noise.

Follow-ups

• How do you detect overfitting in practice? Compare training and validation scores, ideally with cross-validation (question 3), and plot learning curves as the training set grows.
• Can a model overfit the validation set? Yes, if you tune it against that set many times. That's why a final test set stays untouched until the end.

5-fold cross-validation: each row is one training run
Fold:    1      2      3      4      5
Run 1: [val ] [train][train][train][train]
Run 2: [train][val ] [train][train][train]
Run 3: [train][train][val ] [train][train]
Run 4: [train][train][train][val ] [train]
Run 5: [train][train][train][train][val ]

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=cv)
print(scores.round(3), f"mean {scores.mean():.3f} ± {scores.std():.3f}")
# [0.967 0.989 0.978 0.989 0.967] mean 0.978 ± 0.010

model.fit(X_train, y_train)                         # the final model, on all training data
print(f"test {model.score(X_test, y_test):.3f}")    # test 0.982: the test set, used once

The scaler sits inside the pipeline, so in each fold it's fitted on that fold's training part only. Scaling all the data before splitting would leak information from the validation fold into training (question 8).

Train, validation and test sets

When the data is ordered in time, don't shuffle: use TimeSeriesSplit, which always trains on the past and validates on the future, the way the model will actually be used.

Follow-ups

• How do you choose k? 5 or 10 is the usual balance. Larger k means less bias in the estimate but more training runs; leave-one-out (k = n) is rarely worth the cost.
• What is nested cross-validation? An outer loop to estimate performance and an inner loop to tune hyperparameters, so tuning can't flatter the reported score.

Questions to ask before choosing

• Why is it missing? A sensor that failed at random is harmless to fill. Income that's blank because high earners skipped the question is not random, and the blank itself is informative.
• How important is the feature? Check how well the model does with and without it.
• Is the target ever missing? Rows without a target can't be used for training at all; drop those, and never impute the target.

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer

train = pd.DataFrame({"income": [52_000, np.nan, 61_000, 58_000, np.nan, 250_000]})
test = pd.DataFrame({"income": [np.nan, 49_000]})

imputer = SimpleImputer(strategy="median", add_indicator=True)
imputer.fit(train)                          # learn the median from training data only
print(imputer.statistics_)                  # [59500.]
print(pd.DataFrame(imputer.transform(test), columns=imputer.get_feature_names_out()))

    income  missingindicator_income
0  59500.0                      1.0
1  49000.0                      0.0

Why the median: the mean of this training column is 105,250, dragged up by one ₹2.5 lakh income. The median, 59,500, is typical of the rows.

Follow-ups

• What are MCAR, MAR and MNAR? Missing completely at random, missing at random (explained by other columns), and missing not at random (explained by the missing value itself). Only the first is safe to drop without bias.
• Why fit the imputer on training data only? Its median is a statistic of the data. Computing it on everything leaks information about the validation and test rows into training.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

df = pd.DataFrame({
    "city": ["Pune", "Delhi", "Pune", "Chennai"],
    "size": ["small", "large", "medium", "small"],
})

onehot = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
print(pd.DataFrame(onehot.fit_transform(df[["city"]]), columns=onehot.get_feature_names_out()))

ordinal = OrdinalEncoder(categories=[["small", "medium", "large"]])   # the order is stated
print(ordinal.fit_transform(df[["size"]]).ravel())                     # [0. 2. 1. 0.]

print(onehot.transform(pd.DataFrame({"city": ["Kochi"]})))             # [[0. 0. 0.]]: unseen city

   city_Chennai  city_Delhi  city_Pune
0           0.0         0.0        1.0
1           0.0         1.0        0.0
2           0.0         0.0        1.0
3           1.0         0.0        0.0

handle_unknown="ignore" matters in production: a city the model never saw becomes all zeros instead of crashing the prediction.

Follow-ups

• What is the dummy variable trap? With all one-hot columns plus an intercept, one column is always predictable from the others, which breaks an unregularised linear regression. Drop one column (drop="first"), or use regularisation.
• Why not pd.get_dummies? It's fine for exploring, but it doesn't remember the categories it saw, so the test set can end up with different columns. An encoder fitted on training data keeps them consistent.

from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler, StandardScaler

X, y = load_wine(return_X_y=True)        # features range from 0.13 to 1,680
knn = KNeighborsClassifier()

print(f"raw:          {cross_val_score(knn, X, y, cv=5).mean():.2f}")                                  # 0.69
print(f"min-max:      {cross_val_score(make_pipeline(MinMaxScaler(), knn), X, y, cv=5).mean():.2f}")   # 0.95
print(f"standardised: {cross_val_score(make_pipeline(StandardScaler(), knn), X, y, cv=5).mean():.2f}") # 0.95

Unscaled, KNN's distances are dominated by the one feature measured in the hundreds, so the other twelve barely count. Scaling puts them on an equal footing.

Which to choose: standardisation is the usual default. Min-max suits features with natural bounds, such as pixel values from 0 to 255. With strong outliers, RobustScaler uses the median and the interquartile range instead.

Follow-ups

• Should you scale the target? Sometimes, for neural network regression; remember to invert the scaling on the predictions.
• Fit the scaler on what? The training data only, then apply it to validation and test data. A pipeline does this for you.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, weights=[0.97], flip_y=0, random_state=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.3, random_state=1)
print(f"rare class: {y.mean():.1%}")      # rare class: 3.0%

for weights in (None, "balanced"):
    model = LogisticRegression(class_weight=weights, max_iter=1000).fit(X_train, y_train)
    pred = model.predict(X_test)
    print(f"class_weight={weights}: recall {recall_score(y_test, pred):.2f}, precision {precision_score(y_test, pred):.2f}")

proba = LogisticRegression(max_iter=1000).fit(X_train, y_train).predict_proba(X_test)[:, 1]
for threshold in (0.5, 0.2):
    pred = (proba >= threshold).astype(int)
    print(f"threshold {threshold}: recall {recall_score(y_test, pred):.2f}, precision {precision_score(y_test, pred):.2f}")

class_weight=None: recall 0.31, precision 0.78
class_weight=balanced: recall 0.71, precision 0.17
threshold 0.5: recall 0.31, precision 0.78
threshold 0.2: recall 0.49, precision 0.47

Both levers buy recall with precision. Which balance is right is a business decision: how much does a missed case cost compared with a false alarm?

SMOTE lives in the separate imbalanced-learn package, whose own Pipeline resamples inside each cross-validation fold.

Follow-ups

• Why resample only the training data? Validation and test sets must reflect the real class balance, or the scores mean nothing.
• When is imbalance not a problem? When the classes are well separated. Imbalance hurts most when the rare class overlaps the common one.

Leakage you can see in the numbers

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
n = 2000
df = pd.DataFrame({
    "monthly_spend": rng.normal(800, 200, n),
    "support_tickets": rng.poisson(2, n),
})
df["churned"] = (rng.random(n) < 0.1 + 0.08 * df["support_tickets"].clip(upper=6)).astype(int)
# Filled in by the retention team only AFTER a customer has left
df["exit_survey_done"] = df["churned"] * (rng.random(n) < 0.95)

honest = ["monthly_spend", "support_tickets"]
model = RandomForestClassifier(min_samples_leaf=20, random_state=0)
print(cross_val_score(model, df[honest], df["churned"], cv=5, scoring="roc_auc").mean().round(3))
# 0.629: a modest, believable signal
print(cross_val_score(model, df[honest + ["exit_survey_done"]], df["churned"], cv=5, scoring="roc_auc").mean().round(3))
# 0.973: "excellent", and useless, because the feature doesn't exist at prediction time

The checklist

How to say it in the interview: "I'd ask what time each feature becomes available relative to the churn date, look at the feature importances for anything suspiciously strong, and rerun the validation on a later time period."

Follow-ups

• What does a realistic churn model score? It varies, but ROC-AUC in the 0.70s or 0.80s is common; numbers near 1.0 deserve suspicion.
• Can leakage happen after deployment? Yes, if a feature is computed differently in production than in training (training–serving skew). Compute features with the same code in both places.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# Synthetic customer data with some gaps
rng = np.random.default_rng(42)
n = 1000
df = pd.DataFrame({
    "age": rng.integers(21, 60, n).astype(float),
    "monthly_spend": rng.normal(900, 250, n).round(),
    "city": rng.choice(["Pune", "Delhi", "Bengaluru", "Chennai"], n),
    "plan": rng.choice(["basic", "plus", "pro"], n),
})
df.loc[rng.random(n) < 0.1, "age"] = np.nan
df.loc[rng.random(n) < 0.05, "city"] = np.nan
df["churned"] = (((df["plan"] == "basic") & (df["monthly_spend"] < 900)) | (rng.random(n) < 0.1)).astype(int)

numeric = ["age", "monthly_spend"]
categorical = ["city", "plan"]

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))]), categorical),
])
model = Pipeline([("prep", preprocess), ("clf", LogisticRegression(max_iter=1000))])

X, y = df[numeric + categorical], df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

print(f"CV accuracy: {cross_val_score(model, X_train, y_train, cv=5).mean():.3f}")   # 0.886
model.fit(X_train, y_train)
print(f"test accuracy: {model.score(X_test, y_test):.3f}")                            # 0.875

new_customer = pd.DataFrame([{"age": np.nan, "monthly_spend": 750, "city": "Kochi", "plan": "basic"}])
print(model.predict(new_customer))   # [1]: a missing age and an unseen city are handled
print(model[:-1].get_feature_names_out())

['num__age' 'num__monthly_spend' 'cat__city_Bengaluru' 'cat__city_Chennai'
 'cat__city_Delhi' 'cat__city_Pune' 'cat__plan_basic' 'cat__plan_plus'
 'cat__plan_pro']

Why the pipeline matters

• No leakage: inside cross_val_score, each fold refits the imputers, scaler and encoder on that fold's training part only.
• No training–serving skew: the exact same transformations run at prediction time, from one saved object (joblib.dump(model, "churn.joblib")).
• Tuning everything together: a grid search can tune preprocessing choices too, with names like prep__num__impute__strategy.

Follow-ups

• What happens to columns you don't list? ColumnTransformer drops them by default (remainder="drop"); pass remainder="passthrough" to keep them.
• How would you see the transformed data as a DataFrame? Call preprocess.set_output(transform="pandas"). The one-hot encoder must then return a dense array (OneHotEncoder(sparse_output=False)); otherwise scikit-learn raises a ValueError, because pandas output doesn't support sparse data.

A quick numeric check

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(0)
size = rng.uniform(500, 2500, 200)                    # flat size in square feet
price = 3_000 * size + rng.normal(0, 300_000, 200)    # ₹ price: linear, with constant noise

model = LinearRegression().fit(size.reshape(-1, 1), price)
residuals = price - model.predict(size.reshape(-1, 1))
print(round(model.coef_[0]), round(residuals.mean()))   # 2962 0: close to the true ₹3,000 per sq ft

In practice you'd plot residuals against the predictions and look for curves or a funnel shape.

Multicollinearity with VIF

VIF for a feature is 1 / (1 − R²), where R² comes from predicting that feature from all the others:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

def vif(df):
    scores = {}
    for col in df.columns:
        others = df.drop(columns=col)
        r2 = LinearRegression().fit(others, df[col]).score(others, df[col])
        scores[col] = round(1 / (1 - r2), 1)
    return scores

rng = np.random.default_rng(1)
area = rng.uniform(400, 3000, 300)
rooms = (area / 450 + rng.normal(0, 0.4, 300)).round()   # mostly area in disguise
age = rng.uniform(0, 30, 300)
print(vif(pd.DataFrame({"area": area, "rooms": rooms, "age": age})))
# {'area': 14.0, 'rooms': 14.0, 'age': 1.0}

area and rooms carry nearly the same information, so the model can't tell how much credit each deserves.

Follow-ups

• Does multicollinearity hurt predictions? Usually not much; it mainly makes individual coefficients unstable and hard to interpret.
• How is the model fitted? By ordinary least squares: choosing the weights that minimise the sum of squared residuals.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

print(sigmoid(np.array([-4, 0, 4])).round(3))   # [0.018 0.5   0.982]

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X, y)

rows = [29, 215, 542, 224]
proba = model.predict_proba(X[rows])[:, 1]
print(proba.round(3), model.predict(X[rows]))   # [0.021 0.407 0.704 0.98 ] [0 0 1 1]

z = model.decision_function(X[rows])            # the linear score w·x + b
print(np.allclose(sigmoid(z), proba))           # True: probability = sigmoid(linear score)

Odds and log-odds

• A probability of 0.8 means odds of 0.8 / 0.2 = 4 to 1, and log-odds of ln 4 ≈ 1.39.
• The model says log-odds = w·x + b. So a coefficient of 0.7 means one more unit of that feature multiplies the odds by e^0.7 ≈ 2.

Why not mean squared error? With a sigmoid in the model, squared error gives a bumpy loss surface with weak gradients when the model is confidently wrong. Log loss is convex for logistic regression, so gradient descent reliably finds the best weights.

Follow-ups

• More than two classes? Either one-vs-rest (one binary model per class), or multinomial logistic regression with softmax (question 30), which scikit-learn uses by default.
• Why move the threshold? 0.5 isn't sacred. If missing a positive is costly, a lower threshold catches more of them (question 7).
• Is it a linear model? Yes: its decision boundary, where the probability is 0.5, is a straight line (a hyperplane) in feature space.

import numpy as np

def gini(counts):
    p = np.array(counts) / sum(counts)
    return 1 - np.sum(p ** 2)

def entropy(counts):
    p = np.array(counts) / sum(counts)
    p = p[p > 0]                          # 0 × log(0) counts as 0
    return np.sum(p * np.log2(1 / p))

for counts in ([10, 0], [5, 5], [8, 2]):
    print(counts, f"gini {gini(counts):.2f}", f"entropy {entropy(counts):.2f}")

[10, 0] gini 0.00 entropy 0.00
[5, 5] gini 0.50 entropy 1.00
[8, 2] gini 0.32 entropy 0.72

Scoring one split

A node with 5 positives and 5 negatives (Gini 0.5) splits into [4, 0] and [1, 5]. The children's Gini, weighted by size, is 0.4 × 0 + 0.6 × 0.28 ≈ 0.17, so the split reduces impurity by about 0.33. The tree keeps the candidate split with the biggest reduction.

What a small tree learns

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(iris.data, iris.target)
print(export_text(tree, feature_names=iris.feature_names))

|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2

Stopping it from overfitting: max_depth, min_samples_leaf and min_samples_split stop growth early (pre-pruning); ccp_alpha grows the full tree and then cuts back branches that add little (cost-complexity pruning).

Follow-ups

• How does a regression tree split? By the drop in variance (squared error) of the target, and each leaf predicts the mean of its samples.
• Pros and cons of a single tree? Easy to explain, no scaling needed, captures interactions. But it's unstable: a small change in the data can produce a very different tree, which ensembles fix (question 13).

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
for name, model in [
    ("single tree", DecisionTreeClassifier(random_state=0)),
    ("random forest", RandomForestClassifier(n_estimators=300, random_state=0)),
    ("gradient boosting", GradientBoostingClassifier(random_state=0)),
]:
    print(f"{name:18} {cross_val_score(model, X, y, cv=5).mean():.3f}")

single tree        0.917
random forest      0.963
gradient boosting  0.963

Why a random forest beats a single tree: one deep tree overfits in its own particular way. Hundreds of trees trained on different samples, with different features at each split, make different mistakes, and averaging cancels much of the noise while keeping the signal.

Follow-ups

• What makes XGBoost fast and accurate? Regularisation in the tree-building objective, second-order gradient information, efficient split finding, built-in handling of missing values, and parallel computation.
• What is out-of-bag error? Each tree never sees about a third of the rows, so those rows give a free validation estimate (oob_score=True).
• What is stacking? A third ensemble style: a second-level model learns how best to combine the predictions of several different models.

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score

X, _ = make_blobs(n_samples=500, centers=4, cluster_std=1.0, random_state=7)

for k in range(2, 7):
    km = KMeans(n_clusters=k, n_init=10, random_state=7).fit(X)
    print(k, round(km.inertia_), round(silhouette_score(X, km.labels_), 3))

2 15172 0.588
3 4313 0.763
4 920 0.811
5 818 0.694
6 732 0.576

Inertia always falls as k grows, but the big drops stop after k = 4 (the elbow), and the silhouette score, which rewards tight, well-separated clusters, peaks at 4 too.

The algorithm in NumPy

import numpy as np

def kmeans(X, k, iterations=100, seed=0):
    rng = np.random.default_rng(seed)
    centroids = X[rng.choice(len(X), k, replace=False)]        # k random points to start
    for _ in range(iterations):
        distances = ((X[:, None, :] - centroids[None, :, :]) ** 2).sum(axis=2)
        labels = distances.argmin(axis=1)                       # 1. assign to the nearest centroid
        new = np.array([X[labels == j].mean(axis=0) for j in range(k)])   # 2. move to the mean
        if np.allclose(new, centroids):                         # nothing moved: converged
            break
        centroids = new
    return labels, centroids

labels, centroids = kmeans(X, 4)
print(np.bincount(labels))   # [125 125 125 125]: the points in each cluster

KNN vs k-means, a common follow-up

Limitations: k-means assumes roughly round clusters of similar size, is pulled around by outliers, and needs k chosen in advance. DBSCAN finds clusters of any shape and labels noise; Gaussian mixture models allow elongated clusters with soft membership.

Follow-ups

• Why scale first? Distances are dominated by features measured in big units (question 6).
• What does k-means++ do? It picks each new starting centroid with probability proportional to its squared distance from the existing ones, so the starts are spread out and convergence is faster and better.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25, stratify=data.target, random_state=42
)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)               # learns from the training data only
y_pred = model.predict(X_test)

print(f"accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(classification_report(y_test, y_pred, target_names=data.target_names))

accuracy: 0.986
              precision    recall  f1-score   support

   malignant       0.98      0.98      0.98        53
      benign       0.99      0.99      0.99        90

    accuracy                           0.99       143
   macro avg       0.99      0.99      0.99       143
weighted avg       0.99      0.99      0.99       143

Reading the report

• precision: of the cases predicted as this class, the share that were right.
• recall: of the real cases of this class, the share the model found.
• f1-score: the harmonic mean of the two (question 18).
• support: how many test cases belong to the class.
• macro avg treats every class equally; weighted avg weights each class by its support.

Mistakes interviewers look for

• Scaling or encoding before splitting, so the test set influences training.
• Forgetting stratify, so a small test set ends up with too few of one class.
• Reporting only accuracy on imbalanced data (question 19).
• max_iter too low, leaving a ConvergenceWarning unexplained.

Follow-ups

• Why random_state? It makes the split reproducible, so results can be compared and rerun.
• Is one split enough? For a final check, yes; for choosing between models, cross-validation is more reliable (question 3).

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(3)
n = 3000
df = pd.DataFrame({
    "support_tickets": rng.poisson(2, n),
    "monthly_spend": rng.normal(900, 250, n),          # noise: unrelated to churn here
    "tenure_months": rng.integers(1, 60, n),
    "customer_id_digits": rng.integers(0, 10_000, n),  # pure noise with many distinct values
})
logit = 0.6 * df["support_tickets"] - 0.05 * df["tenure_months"] - 1
df["churned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

X, y = df.drop(columns="churned"), df["churned"]
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=3)
model = RandomForestClassifier(n_estimators=200, random_state=3).fit(X_train, y_train)

print(pd.Series(model.feature_importances_, index=X.columns).round(3).sort_values(ascending=False))
perm = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=3)
print(pd.Series(perm.importances_mean, index=X.columns).round(3).sort_values(ascending=False))

customer_id_digits    0.292
monthly_spend         0.289
tenure_months         0.267
support_tickets       0.152
dtype: float64
support_tickets       0.056
tenure_months         0.046
customer_id_digits   -0.000
monthly_spend        -0.001
dtype: float64

The built-in importance ranks the two noise columns highest, because columns with many distinct values give a tree endless split points to overfit with. Permutation importance on held-out data gets it right: only tickets and tenure matter.

Explaining it to the manager: lead with plain language ("customers who raise more support tickets in their first months are the most likely to leave"), show one or two concrete customer examples, and be explicit that this shows what the model uses, not what causes churn. To act on it, test an intervention, such as faster ticket resolution, and measure the effect.

Follow-ups

• Why measure permutation importance on validation data? On training data it also rewards features the model merely memorised.
• What if two features are strongly correlated? Shuffling one barely hurts, because the model leans on the other, so both look unimportant. Group them, or drop one, before interpreting.

from sklearn.metrics import confusion_matrix

actual    = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]   # 1 = has the disease
predicted = [1, 1, 1, 0, 1, 0, 0, 0, 0, 0]

print(confusion_matrix(actual, predicted))
# [[5 1]
#  [1 3]]
tn, fp, fn, tp = confusion_matrix(actual, predicted).ravel()
print(f"TP={tp} FP={fp} TN={tn} FN={fn}")    # TP=3 FP=1 TN=5 FN=1

scikit-learn puts the actual classes in rows and the predicted ones in columns, sorted by label, so the negative class (0) comes first. Check the layout of whatever tool you use; textbooks often draw it the other way round.

The metrics, from those four numbers

Which mistake is worse? For a screening test, a false negative sends a sick person home, while a false positive costs a follow-up test. So screening favours recall, accepting more false alarms (question 18).

Follow-ups

• What about more than two classes? The matrix becomes n × n: each row is a true class, each column a predicted class, and the off-diagonal cells show exactly which classes get confused.
• What are Type I and Type II errors? A false positive and a false negative, in statistics language.

import numpy as np
from sklearn.metrics import f1_score, precision_score, recall_score

actual    = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
predicted = np.array([1, 1, 1, 0, 1, 0, 0, 0, 0, 0])
print(precision_score(actual, predicted), recall_score(actual, predicted), f1_score(actual, predicted))
# 0.75 0.75 0.75

The threshold trades one for the other

scores = np.array([0.95, 0.85, 0.7, 0.45, 0.6, 0.3, 0.2, 0.15, 0.1, 0.05])   # model probabilities
for threshold in (0.8, 0.5, 0.25):
    pred = (scores >= threshold).astype(int)
    p, r = precision_score(actual, pred), recall_score(actual, pred)
    print(f"threshold {threshold}: precision {p:.2f}, recall {r:.2f}")

threshold 0.8: precision 1.00, recall 0.50
threshold 0.5: precision 0.75, recall 0.75
threshold 0.25: precision 0.67, recall 1.00

A strict threshold flags only the surest cases: precise but with many misses. A lenient one catches everything but raises more false alarms.

Why a harmonic mean? It punishes imbalance. A model with precision 1.0 and recall 0.1 has an ordinary average of 0.55, but an F1 of only 0.18, which better reflects how useless it is.

Follow-ups

• What is the F-beta score? A weighted version: F2 counts recall twice as much as precision, F0.5 the reverse.
• Can you have high precision and high recall at once? Only if the model separates the classes well. Otherwise you pick a point on the trade-off curve.

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, average_precision_score, precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20_000, weights=[0.99], flip_y=0, class_sep=0.8, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.3, random_state=7)

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
model = LogisticRegression(class_weight="balanced", max_iter=1000).fit(X_train, y_train)

for name, clf in [("always 'not fraud'", baseline), ("logistic regression", model)]:
    pred = clf.predict(X_test)
    print(f"{name:20} accuracy {accuracy_score(y_test, pred):.3f}  "
          f"recall {recall_score(y_test, pred):.2f}  precision {precision_score(y_test, pred, zero_division=0):.2f}")
print(f"PR-AUC {average_precision_score(y_test, model.predict_proba(X_test)[:, 1]):.2f} (a random model scores about {y_test.mean():.2f})")

always 'not fraud'   accuracy 0.990  recall 0.00  precision 0.00
logistic regression  accuracy 0.946  recall 0.92  precision 0.15
PR-AUC 0.78 (a random model scores about 0.01)

The useful model has lower accuracy than doing nothing, but it catches 92% of fraud. Its precision of 0.15 means about one alert in seven is real, which may or may not be affordable for the review team; the threshold can be tuned to suit.

What to report instead of accuracy

• The confusion matrix, in counts the business understands. For this model: "of 60 frauds, we caught 55, and raised 377 alerts, 322 of them false".
• Recall and precision for the fraud class, at the chosen threshold.
• PR-AUC, to compare models across all thresholds. Its baseline is the fraud rate, not 0.5.
• The money: fraud caught × average loss, minus alerts × review cost.

Follow-ups

• Why PR-AUC rather than ROC-AUC here? ROC's false positive rate divides by the huge number of genuine transactions, so it stays tiny even when false alarms swamp the real frauds. Precision exposes that directly.
• How would you improve the model? Better features (velocity, device, merchant history), class weights or resampling (question 7), and a threshold chosen from the cost trade-off.

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve

actual = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
scores = np.array([0.95, 0.85, 0.7, 0.45, 0.6, 0.3, 0.2, 0.15, 0.1, 0.05])

fpr, tpr, thresholds = roc_curve(actual, scores)
for f, t, th in zip(fpr, tpr, thresholds):
    print(f"threshold {th:>4}: TPR {t:.2f}, FPR {f:.2f}")
print("AUC:", roc_auc_score(actual, scores))

threshold  inf: TPR 0.00, FPR 0.00
threshold 0.95: TPR 0.25, FPR 0.00
threshold  0.7: TPR 0.75, FPR 0.00
threshold  0.6: TPR 0.75, FPR 0.17
threshold 0.45: TPR 1.00, FPR 0.17
threshold 0.05: TPR 1.00, FPR 1.00
AUC: 0.9583333333333334

AUC really is a ranking probability

positives, negatives = scores[actual == 1], scores[actual == 0]
print(np.mean([p > n for p in positives for n in negatives]))   # 0.9583333333333334: the same number

Of the 24 positive–negative pairs, the model ranks the positive higher in 23. The one miss: the positive scored 0.45 sits below a negative scored 0.6.

Follow-ups

• Does a high AUC mean good probabilities? No. AUC only measures ranking. A model can rank perfectly but output badly calibrated probabilities; check a calibration curve if the numbers themselves matter.
• How do you choose the operating threshold? From the costs of each error, or a required recall or precision, using validation data.

import numpy as np
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error

actual = np.array([50, 60, 70, 80, 90])     # delivery times in minutes
model_a = np.array([52, 58, 73, 78, 92])    # small errors everywhere
model_b = np.array([50, 60, 70, 80, 70])    # perfect, except one 20-minute miss

for name, pred in [("A", model_a), ("B", model_b)]:
    mae = mean_absolute_error(actual, pred)
    rmse = root_mean_squared_error(actual, pred)
    print(f"{name}: MAE {mae:.1f}  RMSE {rmse:.1f}  R² {r2_score(actual, pred):.3f}")

A: MAE 2.2  RMSE 2.2  R² 0.975
B: MAE 4.0  RMSE 8.9  R² 0.600

Model B's MAE is less than twice A's, but its RMSE is four times bigger: one big miss dominates a squared metric.

Adjusted R² = 1 − (1 − R²)(n − 1) / (n − p − 1), for n rows and p features. It only rises if a new feature improves the fit more than chance would.

Follow-ups

• Can R² be negative? Yes, on new data, when the model is worse than always predicting the mean.
• Which would you report to a business team? Usually MAE, in their units: "on average we're 2 minutes off".

import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 100)
y = 3 * x + 5 + rng.normal(0, 1, 100)       # the true line: slope 3, intercept 5

def train(lr, steps=1000):
    w, b = 0.0, 0.0
    for _ in range(steps):
        error = w * x + b - y
        w -= lr * 2 * np.mean(error * x)     # the gradient of mean squared error with respect to w
        b -= lr * 2 * np.mean(error)         # ... and with respect to b
    return w, b, np.mean((w * x + b - y) ** 2)

for lr in (0.0001, 0.01, 0.05):
    with np.errstate(over="ignore", invalid="ignore"):   # the large rate overflows
        w, b, mse = train(lr)
    print(f"lr={lr}: w={w:.2f}, b={b:.2f}, MSE={mse:.3g}")

lr=0.0001: w=3.59, b=0.71, MSE=5.23
lr=0.01: w=2.99, b=4.95, MSE=0.941
lr=0.05: w=nan, b=nan, MSE=nan

At 0.0001 it's still far from the answer after 1,000 steps. At 0.01 it finds the true line. At 0.05 every step overshoots further than the last, and the weights explode.

In practice: optimisers such as momentum and Adam adapt the step size per weight and smooth out the noise. A learning-rate schedule (warm-up, then decay) is standard for deep networks, and scaling features helps every variant converge (question 6).

Follow-ups

• What is an epoch? One full pass over the training data: with 10,000 rows and batches of 100, an epoch is 100 updates.
• Does it always find the global minimum? For convex losses such as linear and logistic regression, yes, with a suitable learning rate. Neural network losses aren't convex, but in practice they reach good minima.

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso, LinearRegression, Ridge

# 20 features, only 5 of which actually affect the target
X, y = make_regression(n_samples=200, n_features=20, n_informative=5, noise=10, random_state=0)

for name, model in [("plain", LinearRegression()), ("ridge (L2)", Ridge(alpha=10)), ("lasso (L1)", Lasso(alpha=1))]:
    coef = model.fit(X, y).coef_
    print(f"{name:11} zero weights: {np.sum(np.abs(coef) < 1e-6):2}   largest: {np.abs(coef).max():.1f}")

plain       zero weights:  0   largest: 83.8
ridge (L2)  zero weights:  0   largest: 79.0
lasso (L1)  zero weights: 12   largest: 82.5

Lasso switched off 12 of the 15 useless features. Ridge kept all 20, only slightly smaller.

Elastic Net combines both penalties, which helps when there are groups of correlated features.

Practical notes: the strength λ is alpha in scikit-learn's Ridge and Lasso, and should be tuned with cross-validation (LassoCV, RidgeCV). In LogisticRegression, C is the inverse strength, so a smaller C means stronger regularisation. Always scale features first, so the penalty treats them fairly.

Follow-ups

• Why does L1 give exact zeros? Its penalty has a corner at zero, and the optimum often lands right on it. The L2 penalty is smooth there, so weights approach zero without reaching it.
• Is dropout a form of regularisation? Yes: in neural networks, randomly switching off units during training stops the network relying on any single unit.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)

grid = GridSearchCV(
    RandomForestClassifier(random_state=0),
    param_grid={"max_depth": [3, 6, None], "n_estimators": [100, 300]},
    cv=5,
    scoring="f1",
    n_jobs=-1,                                   # use every CPU core
)
grid.fit(X_train, y_train)                       # 3 × 2 combinations × 5 folds = 30 fits
print(grid.best_params_, f"CV F1 {grid.best_score_:.3f}")
# {'max_depth': None, 'n_estimators': 300} CV F1 0.976
print(f"test F1 {grid.score(X_test, y_test):.3f}")   # test F1 0.958: the held-out set, used once

Random search for bigger spaces

from scipy.stats import randint
from sklearn.model_selection import RandomizedSearchCV

search = RandomizedSearchCV(
    RandomForestClassifier(random_state=0),
    param_distributions={
        "max_depth": randint(2, 20),
        "n_estimators": randint(50, 400),
        "min_samples_leaf": randint(1, 10),
    },
    n_iter=10,                                   # 10 random combinations, not all of them
    cv=5,
    scoring="f1",
    random_state=0,
    n_jobs=-1,
)
search.fit(X_train, y_train)
print(sorted(search.best_params_))   # ['max_depth', 'min_samples_leaf', 'n_estimators']

Parameters vs hyperparameters: parameters are learned from the data (a regression's weights, a tree's split thresholds). Hyperparameters are chosen by you before training (tree depth, learning rate, regularisation strength).

Follow-ups

• Why not tune on the test set? The test score would no longer be an honest estimate: you'd have chosen the model that happened to suit those particular rows.
• What does refit=True do? After the search, it retrains the best combination on all the training data, so grid.predict uses that final model.

import pandas as pd

df = pd.DataFrame(
    {"name": ["Asha", "Ravi", "Priya", "Arjun"],
     "city": ["Pune", "Delhi", "Pune", "Chennai"],
     "salary": [72, 55, 91, 64]},              # ₹ thousands a month
    index=[101, 102, 103, 104],                # employee ids as the index
)

print(df.loc[102, "name"])                     # Ravi: by label
print(df.iloc[1, 0])                           # Ravi: by position
print(df.loc[101:103, "name"].tolist())        # ['Asha', 'Ravi', 'Priya']: end label included
print(df.iloc[0:2, 0].tolist())                # ['Asha', 'Ravi']: end position excluded
print(df.loc[df["city"] == "Pune", ["name", "salary"]])

      name  salary
101   Asha      72
103  Priya      91

Where they drift apart

by_salary = df.sort_values("salary")
print(by_salary.loc[101, "name"])        # Asha: the label still points to Asha
print(by_salary.iloc[0]["name"])         # Ravi: position 0 is now the lowest salary

Changing values: always one loc call

df.loc[df["city"] == "Pune", "salary"] += 5    # works: one call selects rows and column together
print(df["salary"].tolist())                   # [77, 55, 96, 64]

df[df["city"] == "Pune"]["salary"] = 0         # chained: changes a temporary copy, never df
print(df["salary"].tolist())                   # [77, 55, 96, 64]: unchanged

In pandas 3, the chained version also prints a ChainedAssignmentError warning: under copy-on-write, the first [...] always returns a new object, so assigning into it can't affect df.

Follow-ups

• What does df["col"] select? A column by name. df[0:2] slices rows by position: a confusing shortcut, so prefer iloc for clarity.
• How do you reset the index after filtering? df.reset_index(drop=True) renumbers the rows from 0.

import pandas as pd

orders = pd.DataFrame({
    "city": ["Pune", "Pune", "Pune", "Delhi", "Delhi", "Delhi", "Chennai"],
    "product": ["Kettle", "Mixer", "Kettle", "Mixer", "Mixer", "Kettle", "Mixer"],
    "revenue": [1200, 3000, 2500, 3000, 1500, 4000, 2200],
})

totals = orders.groupby(["city", "product"], as_index=False)["revenue"].sum()
best = totals.loc[totals.groupby("city")["revenue"].idxmax()]
print(best)

      city product  revenue
0  Chennai   Mixer     2200
2    Delhi   Mixer     4500
3     Pune  Kettle     3700

Summing first matters: Delhi's single biggest order is a ₹4,000 kettle, but its mixers total ₹4,500.

The sort-and-deduplicate route

best = (totals.sort_values("revenue", ascending=False)
              .drop_duplicates("city")
              .sort_values("city"))
print(best["product"].tolist())   # ['Mixer', 'Mixer', 'Kettle']

Keeping ties

idxmax returns only the first maximum. To keep every product that shares the top total, compare each row with its city's maximum:

top = totals[totals["revenue"] == totals.groupby("city")["revenue"].transform("max")]
print(len(top))   # 3: no ties in this data

Follow-ups

• What does transform do? It returns one value per original row (here, the city's maximum), aligned to the rows, unlike agg, which returns one row per group.
• How would you do this in SQL? With GROUP BY and a window function such as RANK(); see the SQL & DBMS kit, questions 21 and 29.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "age": [25, np.nan, 31, 45, np.nan],
    "income": [52_000, np.nan, 61_000, 250_000, 58_000],
    "city": ["Pune", None, "Delhi", "Pune", "Chennai"],
    "churned": [0, 1, np.nan, 0, 1],
})

print(df.isna().sum())                        # age 2, income 1, city 1, churned 1

df = df.dropna(subset=["churned"])            # no target, no training example
numeric = df.select_dtypes("number").columns.drop("churned")
df[numeric] = df[numeric].fillna(df[numeric].median())
df["churned"] = df["churned"].astype(int)
print(df)

    age    income     city  churned
0  25.0   52000.0     Pune        0
1  35.0   58000.0      NaN        1
3  45.0  250000.0     Pune        0
4  35.0   58000.0  Chennai        1

The median ignores the ₹2.5 lakh outlier; the mean income of those rows would have been 120,000. The missing city is a text column, so it's left for a separate decision: fill it with the most frequent city or an explicit "Unknown".

A pandas 3 trap: inplace on a column

d = pd.DataFrame({"age": [1.0, np.nan]})
d["age"].fillna(0, inplace=True)      # works on a copy; prints a ChainedAssignmentError warning
print(d["age"].tolist())              # [1.0, nan]: unchanged
d["age"] = d["age"].fillna(0)         # assign the result back instead
print(d["age"].tolist())              # [1.0, 0.0]

Under pandas 3's copy-on-write rules, d["age"] is a new object, so changing it in place never reaches d. Older tutorials show the inplace version working.

Follow-ups

• What share of each column is missing? df.isna().mean() gives the fraction per column.
• Why not compute the median on all the data? Because it leaks information from the validation and test rows into training; use SimpleImputer inside a pipeline (question 9).

import pandas as pd

customers = pd.DataFrame({"customer_id": [1, 2, 3, 4], "name": ["Asha", "Ravi", "Priya", "Arjun"]})
orders = pd.DataFrame({"order_id": [101, 102, 103, 104], "customer_id": [1, 1, 3, 9]})

merged = customers.merge(orders, on="customer_id", how="left")
counts = (merged.groupby(["customer_id", "name"], as_index=False)["order_id"].count()
                .rename(columns={"order_id": "orders"}))
print(counts)
print(merged.groupby("customer_id").size().tolist())   # [2, 1, 1, 1]: size() wrongly gives Ravi and Arjun 1

   customer_id   name  orders
0            1   Asha       2
1            2   Ravi       0
2            3  Priya       1
3            4  Arjun       0

What each how keeps

print(customers.merge(orders, on="customer_id", how="outer", indicator=True))

   customer_id   name  order_id      _merge
0            1   Asha     101.0        both
1            1   Asha     102.0        both
2            2   Ravi       NaN   left_only
3            3  Priya     103.0        both
4            4  Arjun       NaN   left_only
5            9    NaN     104.0  right_only

indicator=True adds the _merge column, which is the quickest way to debug a join that returns the wrong number of rows.

Without a merge at all

counts = orders["customer_id"].value_counts().reindex(customers["customer_id"], fill_value=0)
print(counts.tolist())   # [2, 0, 1, 0]

Follow-ups

• merge vs join vs concat? merge joins on columns, like SQL. join is a shortcut that joins on the index. concat stacks DataFrames vertically or side by side without matching keys.
• How do you catch an accidental many-to-many join? Pass validate="one_to_many"; pandas raises an error if the keys aren't unique on the left.
• What's the SQL version? A LEFT JOIN with COUNT(o.order_id); see the SQL & DBMS kit, question 22.

import time
import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 500_000                               # the same idea holds at 5 million, only slower
df = pd.DataFrame({
    "price": rng.integers(100, 5_000, n),
    "discount": rng.integers(0, 50, n),
    "is_member": rng.random(n) < 0.3,
})

def final_price(row):
    return row["price"] - row["discount"] if row["is_member"] else row["price"]

start = time.perf_counter()
slow = df.apply(final_price, axis=1)                  # one Python call per row
apply_time = time.perf_counter() - start

start = time.perf_counter()
fast = np.where(df["is_member"], df["price"] - df["discount"], df["price"])   # whole columns at once
vector_time = time.perf_counter() - start

print((slow.to_numpy() == fast).all())                # True: the same answer
print(f"apply {apply_time:.2f}s, vectorised {vector_time:.4f}s")   # roughly 1 s against under a millisecond

Another vectorised way, with loc

df["final"] = df["price"]
df.loc[df["is_member"], "final"] -= df["discount"]
print((df["final"].to_numpy() == fast).all())         # True

Why NumPy is faster than a Python list

import sys

nums = list(range(1_000_000))
arr = np.arange(1_000_000)
print(sys.getsizeof(nums) + sum(sys.getsizeof(x) for x in nums))   # 36000056: about 36 MB
print(arr.nbytes)                                                   # 8000000: 8 MB

• A list holds pointers to separate Python int objects, each with its own type information and reference count. An array holds raw 8-byte numbers side by side.
• An operation on an array runs one compiled loop with no type checks per element, and modern CPUs can process several packed numbers per instruction.
• df.apply(axis=1) builds a Series for every row and calls your Python function on it, which is why it's the slowest option.

Follow-ups

• What if the logic has several conditions? Use np.select(conditions, choices, default=...), which is the vectorised if / elif / else.
• When is a loop acceptable? When the logic truly depends on the previous row's result, and even then, Numba or Cython can compile it.

import numpy as np

def relu(x):
    return np.maximum(0, x)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def softmax(scores):
    exp = np.exp(scores - scores.max())     # subtracting the max avoids overflow
    return exp / exp.sum()

z = np.array([-2.0, 0.0, 3.0])
print(relu(z))                  # [0. 0. 3.]
print(sigmoid(z).round(3))      # [0.119 0.5   0.953]
print(softmax(z).round(3), softmax(z).sum())   # [0.006 0.047 0.946] 1.0

Two linear layers collapse into one

rng = np.random.default_rng(0)
W1, W2 = rng.normal(size=(4, 3)), rng.normal(size=(2, 4))
x = rng.normal(size=3)
print(np.allclose(W2 @ (W1 @ x), (W2 @ W1) @ x))   # True: one matrix does the work of both

Follow-ups

• What is a dead ReLU? A unit whose input is always negative outputs 0 and gets zero gradient, so it never recovers. Leaky ReLU keeps a small slope below zero.
• Sigmoid or softmax for multi-label tasks, such as tagging a photo "beach" and "sunset"? Sigmoid per label, since the labels aren't mutually exclusive.

An analogy: a relay team loses a race. The coach starts with the final runner's split time, works backwards to see how much each earlier runner's pace contributed, and gives each one a correction proportional to their share of the blame. Backpropagation does that for every weight, in one backward sweep.

A whole network, trained by hand in NumPy

XOR can't be learned by a linear model, so the network needs a hidden layer, and backpropagation to train it:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

rng = np.random.default_rng(1)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)          # XOR

W1, b1 = rng.normal(size=(2, 8)), np.zeros(8)
W2, b2 = rng.normal(size=(8, 1)), np.zeros(1)

for step in range(5000):
    # 1. Forward pass
    h = np.tanh(X @ W1 + b1)
    out = sigmoid(h @ W2 + b2)
    # 2. Backward pass: the chain rule, from the output back to the first layer
    d_out = out - y                        # the loss gradient at the output (log loss + sigmoid)
    d_W2, d_b2 = h.T @ d_out, d_out.sum(axis=0)
    d_h = (d_out @ W2.T) * (1 - h ** 2)    # back through tanh
    d_W1, d_b1 = X.T @ d_h, d_h.sum(axis=0)
    # 3. Update every weight a little, downhill
    for param, grad in ((W1, d_W1), (b1, d_b1), (W2, d_W2), (b2, d_b2)):
        param -= 0.1 * grad

print(out.round(2).ravel())   # [0. 1. 1. 0.]: XOR learned

Frameworks such as PyTorch and TensorFlow do the backward pass for you: loss.backward() computes every gradient automatically (autograd), and optimizer.step() applies the update.

Follow-ups

• Why is it efficient? One backward pass computes the gradients for all the weights, at about the cost of a forward pass, instead of nudging each weight separately to see what happens.
• What can go wrong? Gradients can shrink to nothing or blow up as they travel back through many layers (question 33).

A filter is a small pattern detector, reused everywhere

import numpy as np

image = np.array([
    [0, 0, 0, 9, 9, 9],
    [0, 0, 0, 9, 9, 9],
    [0, 0, 0, 9, 9, 9],
    [0, 0, 0, 9, 9, 9],
], dtype=float)                                   # dark on the left, bright on the right
vertical_edge = np.array([[-1, 0, 1]] * 3, dtype=float)

def convolve(img, kernel):
    kh, kw = kernel.shape
    out = np.zeros((img.shape[0] - kh + 1, img.shape[1] - kw + 1))
    for i in range(out.shape[0]):
        for j in range(out.shape[1]):
            out[i, j] = np.sum(img[i:i + kh, j:j + kw] * kernel)   # the same 9 weights at every position
    return out

print(convolve(image, vertical_edge))
# [[ 0. 27. 27.  0.]
#  [ 0. 27. 27.  0.]]: it fires only where the edge is

In a CNN the filter values aren't hand-written like this; they're learned. A layer of 32 such 3 × 3 filters on a colour image has 3 × 3 × 3 × 32 + 32 = 896 weights. A fully connected layer from a 224 × 224 colour image to just 32 units would need over 4.8 million.

An RNN carries memory from step to step

rng = np.random.default_rng(0)
W_x, W_h, b = rng.normal(size=(4, 3)), rng.normal(size=(4, 4)) * 0.5, np.zeros(4)

h = np.zeros(4)                                   # the memory starts empty
for x_t in rng.normal(size=(5, 3)):               # a sequence of 5 steps, 3 features each
    h = np.tanh(W_x @ x_t + W_h @ h + b)          # the same weights at every step
print(h.shape)                                    # (4,): a summary of the whole sequence

Follow-ups

• Why an LSTM over a plain RNN? Its gates decide what to keep, forget and output, so gradients survive across long sequences; the plain RNN's memory fades quickly (question 33).
• Can a CNN handle text? Yes, with 1-D convolutions over word embeddings, which work well for short-text classification.

The arithmetic of shrinking

for layers in (5, 10, 20):
    print(layers, 0.25 ** layers)   # the best case for sigmoid: its slope is at most 0.25

5 0.0009765625
10 9.5367431640625e-07
20 9.094947017729282e-13

Measured in a real 30-layer network

import numpy as np

rng = np.random.default_rng(0)

def first_layer_gradient(activation, depth=30, width=64):
    x = rng.normal(size=width)
    weights, inputs = [], []
    for _ in range(depth):                        # forward pass through the layers
        scale = np.sqrt(2 / width) if activation == "relu" else np.sqrt(1 / width)
        W = rng.normal(0, scale, (width, width))
        z = W @ x
        weights.append(W)
        inputs.append(z)
        x = np.maximum(0, z) if activation == "relu" else 1 / (1 + np.exp(-z))
    grad = np.ones(width)                         # pretend the loss gradient at the top is 1
    for W, z in zip(reversed(weights), reversed(inputs)):   # backward pass
        if activation == "relu":
            local = (z > 0).astype(float)
        else:
            s = 1 / (1 + np.exp(-z))
            local = s * (1 - s)
        grad = W.T @ (grad * local)
    return np.linalg.norm(grad)

print(f"sigmoid: {first_layer_gradient('sigmoid'):.2e}")   # sigmoid: 1.57e-18
print(f"relu:    {first_layer_gradient('relu'):.2e}")      # relu:    1.94e+01

With sigmoid, the first layer receives a gradient around 10⁻¹⁸: it effectively never learns. With ReLU and He initialisation, the signal survives all 30 layers.

The opposite problem, exploding gradients, happens when the factors are above 1: the gradient grows exponentially and training diverges. Gradient clipping, which caps the gradient's size, is the usual fix, especially for RNNs.

Follow-ups

• How do you spot vanishing gradients? Early layers' weights barely change and their gradient norms are tiny compared with the later layers'.
• Why do transformers suffer less? Residual connections and layer normalisation around every block, and attention links any two positions directly.

TF-IDF can't tell that two words mean the same thing

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = ["I bought a new car", "I bought a new automobile", "I bought a new phone"]
tfidf = TfidfVectorizer().fit_transform(docs)
print(cosine_similarity(tfidf).round(2))

[[1.   0.41 0.41]
 [0.41 1.   0.41]
 [0.41 0.41 1.  ]]

The car sentence is exactly as similar to the phone sentence as to the automobile one: each word is just a separate column.

Embeddings place similar meanings close together

# Hand-made 3-number vectors, for illustration only: real embeddings
# have hundreds of dimensions and are learned from huge amounts of text.
emb = {
    "car":        np.array([0.9, 0.1, 0.0]),
    "automobile": np.array([0.8, 0.2, 0.1]),
    "phone":      np.array([0.1, 0.9, 0.2]),
}

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print(round(cosine(emb["car"], emb["automobile"]), 2))   # 0.98: close in meaning
print(round(cosine(emb["car"], emb["phone"]), 2))        # 0.21: not related

Tokenisation in practice: a sub-word tokeniser (BPE, WordPiece or SentencePiece) might keep "the" as one token but split a rarer word like "tokenisation" into pieces such as "token" and "isation". The exact splits depend on the tokeniser. LLM limits and prices are counted in tokens, and in English a token averages about three-quarters of a word.

Static vs contextual: Word2Vec and GloVe give each word one fixed vector, so "bank" is the same in "river bank" and "bank loan". Transformer models produce contextual embeddings, a different vector for each use, depending on the surrounding words.

Follow-ups

• What does "king − man + woman ≈ queen" show? That word2vec embeddings encode relationships as directions in the vector space.
• Why sub-words rather than whole words? A whole-word vocabulary can't cover every name, typo or new word. Sub-words handle any input with a vocabulary of tens of thousands of tokens.

import numpy as np

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

def self_attention(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    scores = Q @ K.T / np.sqrt(K.shape[1])   # how well each token's query matches every key
    weights = softmax(scores)                # each row sums to 1
    return weights @ V, weights              # a weighted mix of the values

rng = np.random.default_rng(0)
X = rng.normal(size=(3, 4))                  # 3 tokens ("the", "cat", "sat"), 4-number embeddings
Wq, Wk, Wv = (rng.normal(size=(4, 4)) for _ in range(3))

out, weights = self_attention(X, Wq, Wk, Wv)
print(weights.sum(axis=1))                   # [1. 1. 1.]: each token's attention adds up to 1
print(out.shape)                             # (3, 4): a new, context-aware vector per token

The weights would be learned in a real model. After training, the row for "it" in "The cat sat because it was tired" puts much of its weight on "cat".

The other pieces of a transformer

• Multi-head attention: several attention operations side by side, each free to track a different kind of relationship.
• Positional information: attention on its own ignores word order, so positions are added to the embeddings.
• Masking: in a decoder such as GPT, each token may only attend to earlier tokens, so the model can't peek at the word it's predicting.
• Feed-forward layers, residual connections and layer normalisation around each attention block.

Follow-ups

• Why divide by √d? Without it, dot products grow with the vector size, pushing softmax into regions with tiny gradients.
• What's the downside of attention? Its cost grows with the square of the sequence length, which is why long contexts are expensive and why efficient-attention variants exist.

Indexing (once, and again when documents change):
  documents → chunks → embeddings → vector database

Answering (every question):
  question → embedding → top-k most similar chunks → prompt = instructions + chunks + question → LLM → answer with citations

The retrieval step, end to end

TF-IDF stands in for a real embedding model here, so the example runs anywhere; the shape of the code is the same:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

chunks = [
    "Employees get 24 days of paid leave per year, credited monthly.",
    "Laptops are refreshed every three years; raise a ticket with IT.",
    "Unused paid leave of up to 10 days can be carried into the next year.",
    "The office is closed on national holidays listed on the HR portal.",
]
vectorizer = TfidfVectorizer().fit(chunks)      # stands in for the embedding model
index = vectorizer.transform(chunks)            # stands in for the vector database

question = "How many leave days can I carry forward?"
scores = cosine_similarity(vectorizer.transform([question]), index).ravel()
top = scores.argsort()[::-1][:2]                # the two most similar chunks

context = "\n".join(f"[{i}] {chunks[i]}" for i in top)
prompt = (
    "Answer using only the context below and cite the chunk numbers. "
    "If the answer isn't there, say you don't know.\n\n"
    f"Context:\n{context}\n\nQuestion: {question}"
)
print(prompt)

Answer using only the context below and cite the chunk numbers. If the answer isn't there, say you don't know.

Context:
[2] Unused paid leave of up to 10 days can be carried into the next year.
[0] Employees get 24 days of paid leave per year, credited monthly.

Question: How many leave days can I carry forward?

That prompt then goes to the LLM.

What is a vector database? A store for embeddings plus their text and metadata, with fast approximate nearest-neighbour search, so the closest chunks among millions are found in milliseconds. Examples: FAISS (a library), pgvector (a PostgreSQL extension), Pinecone, Weaviate, Qdrant, Milvus and Chroma.

Design choices that decide quality

• Chunking: too small loses context; too large dilutes relevance. A few hundred tokens with some overlap is a common start.
• Retrieval: combine vector search with keyword search (hybrid), since exact terms such as product codes matter, then rerank the candidates.
• Metadata filters: restrict results to the right product, version or language, and to documents the user is allowed to see.

Follow-ups

• What goes wrong in RAG? Usually retrieval: the right chunk isn't found, so the model guesses (question 38).
• RAG or fine-tuning? RAG for knowledge, fine-tuning for behaviour (question 39).

Zero-shot

Classify the sentiment of this review as positive, negative or neutral.
Review: "Delivery was quick but the kettle stopped working after a week."
Sentiment:

Few-shot: show the exact labels and format you want

Classify each support ticket as BILLING, DELIVERY or PRODUCT. Reply with the label only.

Ticket: "I was charged twice for one order." → BILLING
Ticket: "My parcel says delivered but I never got it." → DELIVERY
Ticket: "The mixer's lid doesn't fit." → PRODUCT

Ticket: "Refund still not credited after 10 days." →

Chain-of-thought: ask for the working first

A shop sells pens at ₹12 each, or ₹100 for a pack of 10. Ravi needs 23 pens.
What is the cheapest total? Work through the options step by step, then give
the final answer on its own line as "Answer: ₹<amount>".

(Two packs and three single pens: ₹200 + ₹36 = ₹236.)

What makes any prompt better

• Say who the audience is and what the task is for.
• State constraints explicitly: length, tone, what not to do.
• Fix the output format, ideally a JSON schema the code can check.
• Say what to do when unsure: "If the information isn't provided, say so."
• Keep instructions and data clearly separated, for example with labelled sections.

Follow-ups

• How do you pick few-shot examples? Cover each label and the tricky edge cases, keep them consistent, and vary their order so the model doesn't learn a pattern from position.
• Should you show the chain of thought to users? Usually not: show the final answer, and keep the reasoning for logging or for a separate checking step.

Debug it in order

A cheap automatic guard: check the citations

import re

def check_citations(answer, retrieved_ids):
    cited = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
    return {
        "cited": sorted(cited),
        "no_citation": not cited,
        "not_retrieved": sorted(cited - set(retrieved_ids)),   # cites something it wasn't given
    }

print(check_citations("Unused leave carries over, up to 10 days [2].", [2, 0]))
# {'cited': [2], 'no_citation': False, 'not_retrieved': []}
print(check_citations("Yes, the X200 supports wireless charging.", [5, 7]))
# {'cited': [], 'no_citation': True, 'not_retrieved': []}

An uncited claim isn't proof of hallucination, but it's the right one to send for review or answer with a fallback.

Follow-ups

• Can hallucination be eliminated? Not entirely. You reduce it with grounding, constraints and checks, and design the product so a wrong answer is caught or cheap.
• How would you measure the improvement? A fixed set of real questions with known answers, scored before and after the change (question 40).

The access-control point is often decisive: HR policies visible to managers only must never reach other employees. RAG can filter by the user's permissions before retrieval; a fine-tuned model can't "un-know" what it was trained on.

Why not paste every document into a long context window? Thousands of documents don't fit, every question would pay for all those tokens, and answers get slower and often less accurate when the relevant passage is buried. Retrieval sends only what matters.

Follow-ups

• When would you fine-tune here after all? If answers must follow a strict format, such as citing the policy number and effective date every time, and prompting alone isn't reliable enough.
• How would you keep the index fresh? Re-embed documents when they change, and store each chunk's version and effective date, so answers can say which version they used.

A tiny version of the idea

Real evaluations use rubrics and human review; this keyword check just shows the shape, scoring two prompt versions on the same questions:

eval_set = [
    {"question": "How many paid leave days do I get?", "must_include": ["24"]},
    {"question": "Can I carry leave forward?", "must_include": ["10 days"]},
    {"question": "What is the CEO's salary?", "must_include": ["don't know"]},   # not in the documents
]
answers = {
    "old prompt": ["You get 24 days.", "Yes, without any limit.", "The CEO earns ₹5 crore."],
    "new prompt": ["You get 24 days [0].", "Yes, up to 10 days [2].", "I don't know; that isn't in the policies."],
}

for version, outputs in answers.items():
    passed = [all(fact.lower() in out.lower() for fact in case["must_include"])
              for case, out in zip(eval_set, outputs)]
    failed = [i for i, ok in enumerate(passed) if not ok]
    print(f"{version}: {sum(passed)}/{len(passed)} passed, failed: {failed}")

old prompt: 1/3 passed, failed: [1, 2]
new prompt: 3/3 passed, failed: []

What to measure

Avoiding fooling yourself

• Freeze the set before you start tuning, and add a separate held-out set, so you don't tune the prompt to the test.
• Break results down by category: a better average can hide a new failure on, say, billing questions.
• Small differences can be noise: with 50 questions, one or two flips mean little. Use more cases, or repeated runs, before declaring a winner.
• Then confirm in production: an A/B test on live traffic, watching thumbs-up rates, escalations to human agents and repeat contacts.

Follow-ups

• How do you trust an LLM judge? Give it a precise rubric, have it explain each score, and compare its scores with human ratings on a sample. Swap the order of the two answers to catch position bias.
• What makes a good eval set? Real user questions, not invented ones, covering every intent, with the hard and unanswerable cases deliberately over-represented.