From Python Scripts to ML-Ready Code
Everything you’ve built so far used Python’s built-in lists and dictionaries. The ML ecosystem runs on two libraries that sit underneath almost every model, notebook, and pipeline you’ll touch: NumPy for fast numerical arrays, and pandas for tabular data. This module gets you fluent in both, builds a real preprocessing pipeline, and introduces the core vocabulary of model evaluation.
Plain Python lists are flexible but slow for numerical work — looping over a million numbers in pure Python is dramatically slower than the equivalent operation in a library written for exactly that purpose. NumPy and pandas exist to close that gap, and nearly every ML library (scikit-learn, PyTorch, TensorFlow) is built directly on top of them.
NumPy — Fast Numerical Arrays
A NumPy array (ndarray) looks similar to a list but behaves very differently. Every element must be the same type (usually float or int), which lets NumPy store and process the whole array using fast, compiled code instead of Python’s slower general-purpose loops. Operations apply to the entire array at once — no explicit loop required.
The shape of an array describes its dimensions — (5,) is a flat list of 5 numbers, (3, 4) is 3 rows by 4 columns. Almost every NumPy bug a beginner hits traces back to a shape mismatch, so checking .shape is the single most useful debugging habit you can build.
The core idea that makes NumPy fast and pleasant to use is vectorization: instead of writing for x in values: result.append(x * 2), you write values * 2 and NumPy applies the operation to every element internally, in compiled code. This isn’t just shorter — it’s typically 10–100x faster for large arrays.
pandas — Tabular Data with Labels
A pandas DataFrame is a 2D table with named columns and a labelled index — conceptually similar to the list-of-dicts pattern from M3, but with far more built-in functionality for filtering, grouping, joining, and analysing data. A Series is a single labelled column — think of a DataFrame as a dictionary of Series sharing the same index.
| name | department | salary | |
|---|---|---|---|
| 0 | Priya Sharma | IT | 85000 |
| 1 | Rahul Mehta | IT | 92000 |
| 2 | Anita Patel | Finance | 78000 |
If you’ve worked through M3 (Data Structures) and M5 (File Handling), pandas will feel immediately familiar — df[“salary”] is conceptually the same idea as accessing a column across a list of dicts, just with vastly more built-in power: df.groupby(“department”).mean() replaces several lines of manual dictionary accumulation with one line.
Data Preprocessing — Cleaning Before Modelling
Raw data is almost never ready for a model. Missing values, inconsistent scales, and categorical (text) columns all need handling before any algorithm can use the data sensibly. This is typically where an ML engineer spends the most time — far more than on the modelling step itself.
Missing values can be dropped (df.dropna()) or filled in (df.fillna()) with a sensible value — the mean, median, or a placeholder, depending on the column. Encoding converts text categories into numbers a model can use — one-hot encoding turns a “department” column with values IT/Finance/HR into three binary columns. Scaling puts numeric features on comparable ranges (e.g. 0–1) so that a feature measured in thousands doesn’t dominate one measured in single digits purely because of its scale.
Train/Test Split — Why You Never Evaluate on Training Data
A model evaluated on the same data it learned from will look deceptively good — it may have simply memorised the answers rather than learned a generalisable pattern. The standard practice is to split your data into a training set (used to fit the model) and a test set (held back, used only to evaluate how well the model performs on data it has never seen).
A typical split is 80% train / 20% test, though this varies by dataset size and problem. The critical rule: the test set must never influence training in any way — not the model fitting, not the scaling parameters, not feature selection decisions. Any leakage from test to train makes your evaluation numbers optimistic and unreliable in production.
Model Evaluation — Reading the Right Metric
“My model is 95% accurate” sounds great until you learn the dataset is 95% one class — a model that always predicts that one class would score exactly as well while learning nothing. Knowing which metric actually answers your question is as important as the modelling itself.
A spam filter that misses a few spam emails (low recall) is mildly annoying. A fraud detector that misses real fraud (low recall) is expensive. A medical test with too many false alarms (low precision) causes unnecessary panic and follow-up costs. The right metric depends entirely on what mistake is more costly in your specific context — there is no universally “best” metric.
NumPy — creating and inspecting arrays
import numpy as np
# Creating arrays
a = np.array([12, 45, 8, 33, 19]) # from a list — 1D
b = np.array([[1, 2, 3], [4, 5, 6]]) # from nested lists — 2D
zeros = np.zeros((3, 4)) # 3x4 array of zeros
ones = np.ones(5) # [1. 1. 1. 1. 1.]
ranged = np.arange(0, 10, 2) # [0 2 4 6 8] — like range() but returns an array
# Inspecting arrays
a.shape # (5,) — dimensions
b.shape # (2, 3) — 2 rows, 3 columns
a.dtype # dtype('int64') — the data type of every element
a.size # 5 — total number of elements
# Vectorized operations — no loop needed
a * 2 # [24 90 16 66 38] — every element doubled
a + 10 # [22 55 18 43 29] — every element +10
a > 20 # [False True False True False] — boolean array
a[a > 20] # [45 33] — boolean indexing: only elements where condition is True
# Aggregate functions
a.mean() # 23.4
a.std() # standard deviation
a.min(), a.max() # 8, 45
a.sum() # 117
# Slicing — works like list slicing, but supports multiple dimensions
b[0] # [1 2 3] — first row
b[:, 1] # [2 5] — second column (all rows, column index 1)
b[0, 2] # 3 — single element, row 0 column 2pandas — reading, inspecting, and selecting data
import pandas as pd
# Creating a DataFrame
df = pd.DataFrame({
"name": ["Priya", "Rahul", "Anita"],
"department": ["IT", "IT", "Finance"],
"salary": [85000, 92000, 78000],
})
# Reading from a file (most common in practice)
df = pd.read_csv("employees.csv")
df = pd.read_json("employees.json")
# Inspecting
df.shape # (3, 3) — rows, columns
df.columns # Index(['name', 'department', 'salary'])
df.dtypes # data type of each column
df.head(5) # first 5 rows — quick sanity check
df.describe() # summary stats (mean, std, min, max...) for numeric columns
df.info() # column types and non-null counts in one view
# Selecting columns
df["salary"] # a single column — returns a Series
df[["name", "salary"]] # multiple columns — returns a DataFrame
# Filtering rows (boolean indexing — just like NumPy)
df[df["salary"] > 80000]
df[(df["department"] == "IT") & (df["salary"] > 80000)]
# Adding / modifying columns
df["bonus"] = df["salary"] * 0.1 # vectorized — applies to every row
df["band"] = df["salary"].apply(lambda s: "Senior" if s > 90000 else "Mid")
# Grouping and aggregating
df.groupby("department")["salary"].mean()
df.groupby("department").agg({"salary": ["mean", "count"]})
# Sorting
df.sort_values("salary", ascending=False)Handling missing values
# Detecting missing values
df.isna() # DataFrame of True/False — True where value is missing
df.isna().sum() # count of missing values per column
# Dropping missing values
df.dropna() # removes any row with at least one missing value
df.dropna(subset=["salary"]) # only drops rows where 'salary' specifically is missing
# Filling missing values
df["salary"].fillna(0) # fill with a fixed value
df["salary"].fillna(df["salary"].mean()) # fill with the column mean — common for numeric data
df["department"].fillna("Unknown") # fill with a placeholder — common for categoriesEncoding categories and scaling features
# One-hot encoding — turns categories into binary columns
pd.get_dummies(df["department"])
# produces columns: department_Finance, department_IT (1/0 values)
df_encoded = pd.get_dummies(df, columns=["department"]) # applies to the whole frame
# Min-max scaling manually (scales values to a 0-1 range)
col = df["salary"]
df["salary_scaled"] = (col - col.min()) / (col.max() - col.min())
# Using scikit-learn's scalers (the standard tool in practice)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df[["salary_scaled"]] = scaler.fit_transform(df[["salary"]])
# StandardScaler centres data to mean 0, std-dev 1 (the "normalise" pattern from M7)Train/test split and basic evaluation metrics
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, precision_score,
recall_score, f1_score, confusion_matrix)
# Splitting features (X) and target (y) into train/test sets
X = df[["cpu", "mem", "disk"]] # features — what the model learns FROM
y = df["failed"] # target — what the model learns to PREDICT
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
# random_state fixes the shuffle so results are reproducible run to run
# After training a model and getting predictions (y_pred)...
accuracy_score(y_test, y_pred)
precision_score(y_test, y_pred)
recall_score(y_test, y_pred)
f1_score(y_test, y_pred)
confusion_matrix(y_test, y_pred)
# confusion_matrix returns a 2x2 grid: [[true_neg, false_pos], [false_neg, true_pos]]These five examples build on each other in sequence, following the exact shape of a real ML workflow — they’re deliberately ordered as load → explore → clean → split → evaluate, the same pipeline you’ll use on virtually every project.
The first step of any ML project — load a CSV and understand its shape, types, and quality before doing anything else. Skipping this step is the most common cause of confusing bugs three steps later.
import pandas as pd
# Simulating a realistic server-incident dataset
data = {
"server": ["web-01", "web-02", "db-01", "db-02", "api-01", "cache-01", "web-03"],
"cpu_avg": [45, 88, 62, 91, 40, 35, None],
"mem_avg": [60, 72, 91, 85, 55, 48, 66],
"disk_pct": [55, 40, 88, 92, 30, 25, 61],
"environment": ["prod", "prod", "prod", "prod", "staging", "staging", None],
"incident": [0, 1, 1, 1, 0, 0, 0],
}
df = pd.DataFrame(data)
print("Shape:", df.shape)
print("\nFirst 3 rows:")
print(df.head(3))
print("\nMissing values per column:")
print(df.isna().sum())
print("\nSummary statistics:")
print(df[["cpu_avg", "mem_avg", "disk_pct"]].describe().round(1))
print("\nIncident rate by environment:")
print(df.groupby("environment")["incident"].mean())First 3 rows:
server cpu_avg mem_avg disk_pct environment incident
0 web-01 45.0 60 55 prod 0
1 web-02 88.0 72 40 prod 1
2 db-01 62.0 91 88 prod 1
Missing values per column:
server 0
cpu_avg 1
mem_avg 0
disk_pct 0
environment 1
incident 0
Incident rate by environment:
environment
prod 0.75
staging 0.00
Continues directly from Example 1’s dataset, fixing the missing values it surfaced and converting the text “environment” column into a numeric form a model can actually use.
# ── Continuing from the df built in Example 1 ──
# Fill missing numeric value with the column mean
df["cpu_avg"] = df["cpu_avg"].fillna(df["cpu_avg"].mean())
# Fill missing category with an explicit placeholder, never silently drop it
df["environment"] = df["environment"].fillna("unknown")
print("After cleaning — missing values:")
print(df.isna().sum())
print()
# One-hot encode the environment column
df_encoded = pd.get_dummies(df, columns=["environment"])
print("Columns after one-hot encoding:")
print(list(df_encoded.columns))
print()
print(df_encoded[["server", "environment_prod", "environment_staging", "environment_unknown"]])server 0
cpu_avg 0
mem_avg 0
disk_pct 0
environment 0
incident 0
Columns after one-hot encoding:
[‘server’, ‘cpu_avg’, ‘mem_avg’, ‘disk_pct’, ‘incident’, ‘environment_prod’, ‘environment_staging’, ‘environment_unknown’]
server environment_prod environment_staging environment_unknown
0 web-01 True False False
1 web-02 True False False
…
Demonstrates why scaling matters using a concrete, visible example, then implements min-max scaling using vectorized NumPy operations rather than a manual loop — the performance and clarity benefit of vectorization made tangible.
import numpy as np
# Two features on very different scales
disk_gb = np.array([120, 450, 80, 900, 300]) # range: tens to hundreds
response_ms = np.array([12, 45, 8, 90, 30]) # range: single to double digits
print("Before scaling:")
print(f" disk_gb range: {disk_gb.min()}-{disk_gb.max()}")
print(f" response_ms range: {response_ms.min()}-{response_ms.max()}")
print(" → disk_gb would dominate any distance-based calculation purely due to scale")
def min_max_scale(arr):
"""Scale a NumPy array to the 0-1 range. Fully vectorized — no loop."""
return (arr - arr.min()) / (arr.max() - arr.min())
disk_scaled = min_max_scale(disk_gb)
response_scaled = min_max_scale(response_ms)
print("\nAfter min-max scaling (both now 0-1):")
print(f" disk_gb scaled: {np.round(disk_scaled, 2)}")
print(f" response_ms scaled: {np.round(response_scaled, 2)}")disk_gb range: 80-900
response_ms range: 8-90
→ disk_gb would dominate any distance-based calculation purely due to scale
After min-max scaling (both now 0-1):
disk_gb scaled: [0.05 0.45 0. 1. 0.27]
response_ms scaled: [0.05 0.45 0. 1. 0.27]
Splits a dataset into training and test portions using scikit-learn, then demonstrates the single most important rule of preprocessing: any scaler must be fit only on training data, then merely applied to test data — never fit on the test set.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
df = pd.DataFrame({
"cpu_avg": [45, 88, 62, 91, 40, 35, 70, 55, 82, 30],
"mem_avg": [60, 72, 91, 85, 55, 48, 66, 58, 79, 42],
"disk_pct": [55, 40, 88, 92, 30, 25, 60, 35, 70, 20],
"incident": [0, 1, 1, 1, 0, 0, 1, 0, 1, 0],
})
X = df[["cpu_avg", "mem_avg", "disk_pct"]]
y = df["incident"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42
)
print(f"Train set: {X_train.shape[0]} rows Test set: {X_test.shape[0]} rows")
# CORRECT: fit the scaler on training data only, then transform both
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # learns mean/std FROM training data
X_test_scaled = scaler.transform(X_test) # APPLIES those same learned values — does not re-learn
print(f"\nTraining set mean (should be ~0 after scaling): {X_train_scaled.mean():.4f}")
print(f"Test set mean (won't be exactly 0 — scaled using TRAIN's mean/std, not its own): "
f"{X_test_scaled.mean():.4f}")Training set mean (should be ~0 after scaling): 0.0000
Test set mean (won’t be exactly 0 — scaled using TRAIN’s mean/std, not its own): 0.2147
Takes a set of true labels and predicted labels (as if from a trained model) and computes accuracy, precision, recall, F1, and a confusion matrix — then shows concretely why accuracy alone would have been misleading on this particular dataset.
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
f1_score, confusion_matrix)
# Imbalanced scenario: rare server incidents, mostly "no incident" (0)
y_true = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1] # 8 "no incident", 2 "incident"
# Model A: predicts "no incident" for everything — lazy, but scores well on accuracy
y_pred_lazy = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
# Model B: actually tries to catch incidents, makes one mistake
y_pred_real = [0, 0, 0, 0, 0, 0, 1, 0, 1, 1]
def evaluate(name, y_true, y_pred):
print(f"--- {name} ---")
print(f"Accuracy : {accuracy_score(y_true, y_pred):.2f}")
print(f"Precision: {precision_score(y_true, y_pred, zero_division=0):.2f}")
print(f"Recall : {recall_score(y_true, y_pred, zero_division=0):.2f}")
print(f"F1 Score : {f1_score(y_true, y_pred, zero_division=0):.2f}")
print(f"Confusion matrix:\n{confusion_matrix(y_true, y_pred)}\n")
evaluate("Model A — predicts everything is fine", y_true, y_pred_lazy)
evaluate("Model B — actually tries to detect incidents", y_true, y_pred_real)Accuracy : 0.80
Precision: 0.00
Recall : 0.00
F1 Score : 0.00
Confusion matrix:
[[8 0]
[2 0]]
— Model B — actually tries to detect incidents —
Accuracy : 0.90
Precision: 0.67
Recall : 1.00
F1 Score : 0.80
Confusion matrix:
[[7 1]
[0 2]]
You’ll need numpy, pandas, and scikit-learn installed (pip install numpy pandas scikit-learn). Work through these in order — each one assumes comfort with the previous.
📋 Server Incident Prediction Pipeline
Build a script called incident_pipeline.py that takes raw, messy server metrics and prepares them fully for modelling — then evaluates a (manually supplied) set of predictions, bringing together every stage from this module’s example sequence.
- Create server_metrics.csv with at least 15 rows and columns: server, cpu_avg, mem_avg, disk_pct, environment, incident (0/1 target). Deliberately include at least 3 missing numeric values and 1 missing category value. Make the incident column imbalanced (mostly 0s, a handful of 1s) to mirror real-world rarity of actual incidents.
- load_and_explore(path) — reads the CSV, prints shape, missing-value counts per column, and .describe() for the numeric columns. Returns the loaded DataFrame.
- clean_data(df) — fills missing numeric values with their column mean, fills the missing category with “unknown”, and returns the cleaned DataFrame. Must not modify the caller’s original DataFrame in place — return a new one (hint: .copy(), echoing the same principle from M7’s assignment).
- encode_and_split(df) — one-hot encodes the environment column, separates features (X) from the target (y, the incident column), and returns a train/test split with test_size=0.25 and a fixed random_state for reproducibility.
- scale_features(X_train, X_test) — fits a StandardScaler on X_train only, then transforms both X_train and X_test using that same fitted scaler. Returns both scaled arrays.
- evaluate(y_test, y_pred) — prints accuracy, precision, recall, F1, and the confusion matrix. Since this assignment doesn’t require training an actual model, supply a manually written y_pred list matching the length of your test set and pass it through this function to confirm it works correctly.
These questions probe the judgment calls that separate someone who can run sklearn functions from someone who understands what those functions are actually protecting against.