Tutorial 7: Descriptive Statistics with Python

By /

Introduction

Descriptive statistics helps us summarize and understand data using numbers, so we don’t have to examine every individual value one by one. It answers questions like: what’s the average? What’s the middle value? How spread out is the data? What’s the highest or lowest value? Are two variables related?

Raw Data → Descriptive Statistics → Summary Numbers → Better Understanding

You’ll learn: mean, median, mode, range, variance, standard deviation, percentiles, quartiles, correlation, and covariance.

1. Create Sample Data

import pandas as pd
import numpy as np

salary = [45000, 50000, 52000, 55000, 60000, 62000, 65000, 68000, 70000, 75000]

df = pd.DataFrame({"Salary": salary})
df

2. Mean

The mean is the average value.

Formula: Mean = Sum of all values ÷ Number of values

df["Salary"].mean()

Manually:

sum(salary) / len(salary)

Example: for salaries 50,000 / 60,000 / 70,000 → Mean = (50,000 + 60,000 + 70,000) ÷ 3 = 60,000.

Use the mean when data is relatively balanced and there are no extreme outliers.

3. Median

The median is the middle value when data is sorted.

df["Salary"].median()

Example: 10, 20, 30, 40, 50 → Median = 30. For an even count, e.g. 10, 20, 30, 40 → Median = (20 + 30) ÷ 2 = 25.

Mean vs. Median: Consider 30,000 / 40,000 / 50,000 / 60,000 / 1,000,000. That one extreme value drags the mean way up. In cases like this, the median better represents a “typical” value — this is exactly why, for example, median household income (not mean) is usually reported in the news.

4. Mode

The mode is the most frequently occurring value.

data = [10, 20, 20, 30, 40]
series = pd.Series(data)
series.mode()

Output: 20

Mode is useful for: most common product, most common category, most frequent customer type, most common survey response.

5. Minimum and Maximum

df["Salary"].min()
df["Salary"].max()

6. Range

The range is the difference between the maximum and minimum.

Formula: Range = Maximum − Minimum

salary_range = df["Salary"].max() - df["Salary"].min()
salary_range

Example: Maximum = 75,000, Minimum = 45,000 → Range = 30,000.

7. Variance

Variance measures how far data values spread out from the mean. A small variance means values cluster close to the mean; a large variance means they’re widely spread.

df["Salary"].var()

Concept: Values → Compare with Mean → Measure Differences → Calculate Variance. Note that variance is expressed in squared units — if salary is in rupees, variance is in “rupees squared,” which is part of why it’s less intuitive to interpret directly than standard deviation.

8. Standard Deviation

Standard deviation is the most commonly used measure of spread, precisely because it fixes variance’s “squared units” problem.

df["Salary"].std()

Relationship: Standard Deviation = √Variance

If Mean Salary = ₹60,000 and Standard Deviation = ₹10,000, then salaries generally vary around the mean by roughly ₹10,000 — much easier to interpret than “100,000,000 rupees-squared.”

9. Percentiles

A percentile tells you the value below which a given percentage of the data falls. (The 50th percentile is the same as the median.)

df["Salary"].quantile(0.25)   # 25th percentile
df["Salary"].quantile(0.50)   # 50th percentile
df["Salary"].quantile(0.75)   # 75th percentile
df["Salary"].quantile(0.90)   # 90th percentile

Example: if the 75th percentile of salary is ₹70,000, that means roughly 75% of salaries are less than or equal to ₹70,000.

10. Quartiles

Quartiles divide the data into four equal parts:

Minimum ── Q1 ── Q2 ── Q3 ── Maximum
  0%      25%    50%    75%    100%

Where Q1 = 25th percentile, Q2 = median (50th percentile), Q3 = 75th percentile.

Q1 = df["Salary"].quantile(0.25)
Q2 = df["Salary"].quantile(0.50)
Q3 = df["Salary"].quantile(0.75)

print("Q1:", Q1)
print("Q2:", Q2)
print("Q3:", Q3)

11. Interquartile Range (IQR)

The IQR measures the spread of the middle 50% of the data.

Formula: IQR = Q3 − Q1

IQR = Q3 - Q1
IQR

IQR is especially useful for detecting outliers, since it ignores the extreme top and bottom quarters.

12. Detect Outliers Using IQR

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

outliers = df[
    (df["Salary"] < lower_bound) |
    (df["Salary"] > upper_bound)
]
outliers

This “1.5 × IQR” rule is a widely-used convention (also what draws the whiskers on a box plot) for flagging values that fall unusually far from the rest of the data.

13. Pandas describe()

df["Salary"].describe()

This single command returns count, mean, std, min, 25%, 50%, 75%, and max — essentially every measure from this chapter, computed at once.

For the entire DataFrame:

df.describe()

14. Correlation

Correlation measures the relationship between two variables — for example: do employees with more experience generally earn higher salaries?

data = {
    "Experience": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    "Salary": [30000, 35000, 40000, 45000, 50000, 58000, 65000, 72000, 80000, 90000]
}
df = pd.DataFrame(data)

df["Experience"].corr(df["Salary"])

15. Understanding Correlation Values

-1 ────────── 0 ────────── +1
Negative    No relation    Positive

Positive correlation: both variables tend to increase together (e.g., experience ↑, salary ↑). Negative correlation: one increases while the other decreases (e.g., price ↑, demand ↓). Zero correlation: no clear linear relationship at all.

16. Correlation Visualization

import seaborn as sns
import matplotlib.pyplot as plt

correlation = df.corr()

sns.heatmap(correlation, annot=True)
plt.title("Correlation Matrix")
plt.show()

17. Important: Correlation Does Not Mean Causation

Suppose we observe that ice cream sales and swimming accidents both rise together — they may show a positive correlation, but ice cream doesn’t cause swimming accidents. A third factor — hot weather — likely drives both independently.

        Hot Weather
        ↙         ↘
Ice Cream Sales   Swimming Accidents

Correlation shows a relationship, but it does not prove that one variable causes another. This is one of the most important — and most often violated — principles in all of statistics and data journalism.

Summary of Statistical Measures

MeasureWhat It Tells Us
MeanAverage value
MedianMiddle value
ModeMost frequent value
Minimum / MaximumSmallest / largest value
RangeOverall spread
VarianceAverage squared spread
Standard DeviationTypical spread from the mean
PercentilePosition of a value in the dataset
QuartileDivides data into four sections
IQRSpread of the middle 50%
CorrelationRelationship between variables

Practice Exercise

Using the Titanic dataset (df = sns.load_dataset("titanic")), calculate: average passenger age, median passenger age, most common passenger class, standard deviation of ticket fare, 25th/75th percentiles of fare, correlation between age and fare, and correlation between passenger class and fare. Then create a histogram of age, a box plot of fare, and a correlation heatmap.

Scroll to Top