Tutorial 6: Exploratory Data Analysis (EDA)

Introduction

Exploratory Data Analysis (EDA) is the process of investigating a dataset thoroughly before drawing conclusions or building models. It’s where everything from earlier chapters — Pandas, Matplotlib, and Seaborn — comes together into a single practical workflow.

Dataset → Load → Understand → Clean → Explore Statistics → Visualize Patterns → Find Insights

By the end of this chapter, you’ll be able to load a real dataset, understand its structure, check data quality, handle missing values, explore numerical and categorical columns, visualize distributions and relationships, and identify useful insights.

---

1. The Dataset

We’ll use the classic Titanic dataset, containing information about passengers aboard the Titanic — a popular dataset for practicing EDA because it mixes numerical, categorical, and missing data.

Column	Meaning
survived	Whether the passenger survived
pclass	Passenger class
sex	Gender
age	Age
fare	Ticket fare
embarked	Port of embarkation

We can load it directly using Seaborn’s built-in sample datasets:

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

df = sns.load_dataset("titanic")
df

---

2. Understand the Dataset

df.head()
df.tail()
df.shape

Example output for .shape: (891, 15) → 891 rows, 15 columns.

df.columns
df.info()

df.info() is one of the most important EDA commands — it reveals column names, the number of non-null values, data types, and where missing data lives, all in one call.

---

3. Statistical Summary

df.describe()

This gives count, mean, standard deviation, minimum, 25th percentile, median, 75th percentile, and maximum — but only for numerical columns.

For categorical columns:

df.describe(include="object")

---

4. Check Missing Values

df.isnull().sum()

Visualize where missing values are located:

sns.heatmap(df.isnull(), cbar=False)
plt.title("Missing Values")
plt.show()

Why is this useful? Instead of only seeing a list like “age: 177 missing, deck: 688 missing,” the heatmap gives you a visual map of exactly where the gaps are — which is often much faster to interpret, especially on wide datasets.

---

5. Data Cleaning

Check duplicate records:

df.duplicated().sum()
df = df.drop_duplicates()

Fill missing age values (using the median):

df["age"] = df["age"].fillna(df["age"].median())

Fill missing embarkation values (using the mode):

df["embarked"] = df["embarked"].fillna(df["embarked"].mode()[0])

mode() returns the most frequently occurring value — useful for categorical columns where “average” doesn’t make sense, but “most common” does.

---

6. Analyze Survival

df["survived"].value_counts()

Visualize it:

sns.countplot(data=df, x="survived")
plt.title("Survival Count")
plt.show()

In this dataset, 0 means “did not survive” and 1 means “survived.”

---

7. Survival by Gender

sns.countplot(data=df, x="sex", hue="survived")
plt.title("Survival by Gender")
plt.show()

This lets us directly compare survival counts between male and female passengers.

---

8. Survival Rate by Gender

Rather than raw counts, calculate the rate:

df.groupby("sex")["survived"].mean()

This produces a value between 0 and 1. To express as a percentage:

survival_by_gender = df.groupby("sex")["survived"].mean() * 100
survival_by_gender

Insight: we can now directly compare the percentage of passengers who survived, broken down by gender.

---

9. Survival by Passenger Class

sns.countplot(data=df, x="pclass", hue="survived")
plt.title("Survival by Passenger Class")
plt.show()

Calculate survival rate:

df.groupby("pclass")["survived"].mean()

This allows us to investigate whether passenger class was associated with survival.

---

10. Age Distribution

sns.histplot(data=df, x="age", kde=True)
plt.title("Age Distribution")
plt.show()

This shows the most common age ranges, the overall shape of the distribution, and whether data clusters around certain ages.

---

11. Age Distribution by Survival

sns.histplot(data=df, x="age", hue="survived", kde=True)
plt.title("Age Distribution by Survival")
plt.show()

This overlays the age distributions of survivors and non-survivors, so you can compare them directly.

---

12. Fare Distribution

sns.histplot(data=df, x="fare", kde=True)
plt.title("Fare Distribution")
plt.show()

Because fares vary widely (a few very expensive tickets can skew a histogram), a box plot is often clearer:

sns.boxplot(data=df, x="fare")
plt.title("Fare Distribution")
plt.show()

---

13. Relationship Between Age and Fare

sns.scatterplot(data=df, x="age", y="fare", hue="survived")
plt.title("Age vs Fare")
plt.show()

We’re now examining three variables at once — age, fare, and survival — an example of multivariable exploration.

---

14. Correlation Analysis

numeric_columns = ["survived", "pclass", "age", "sibsp", "parch", "fare"]
correlation = df[numeric_columns].corr()

sns.heatmap(correlation, annot=True)
plt.title("Correlation Heatmap")
plt.show()

This reveals which numerical variables tend to move together.

---

15. Grouped Analysis

Analyze survival by both gender and passenger class simultaneously:

df.groupby(["sex", "pclass"])["survived"].mean()

This is far more detailed than looking at either variable alone. Visualize it:

sns.barplot(data=df, x="pclass", y="survived", hue="sex")
plt.title("Survival Rate by Class and Gender")
plt.show()

---

16. Build an EDA Summary

At the end of any EDA project, summarize your findings clearly:

Dataset: Contains passenger information with both numerical and categorical variables.

Data Quality: Some columns contain missing values; duplicates were checked; missing numerical values were handled using the median.

Exploration: Survival rates were compared by gender and by passenger class; age and fare distributions were analyzed; relationships between numerical variables were explored.

Insights: Survival outcomes differed meaningfully across passenger groups — passenger class and gender emerge as useful variables for further analysis (e.g., in a predictive model).

---

Complete EDA Workflow

Load Dataset (pd.read_csv())
      ↓
Understand Data (head(), info(), shape, columns)
      ↓
Check Data Quality (Missing Values, Duplicates, Incorrect Values)
      ↓
Clean Data (fillna(), drop_duplicates())
      ↓
Analyze Data (groupby(), mean(), count())
      ↓
Visualize Data (Matplotlib, Seaborn)
      ↓
Find Insights (Answer Questions)

---

Complete EDA Code

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
df = sns.load_dataset("titanic")

# 1. Understand the dataset
print("Shape:")
print(df.shape)
print("\nColumns:")
print(df.columns)
print("\nFirst 5 Rows:")
display(df.head())
print("\nDataset Information:")
df.info()

# 2. Check missing values
print("\nMissing Values:")
print(df.isnull().sum())

# 3. Remove duplicates
df = df.drop_duplicates()

# 4. Handle missing values
df["age"] = df["age"].fillna(df["age"].median())
df["embarked"] = df["embarked"].fillna(df["embarked"].mode()[0])

# 5. Statistical summary
print("\nStatistical Summary:")
display(df.describe())

# 6. Survival analysis
print("\nSurvival Count:")
print(df["survived"].value_counts())
print("\nSurvival by Gender:")
display(df.groupby("sex")["survived"].mean())
print("\nSurvival by Class:")
display(df.groupby("pclass")["survived"].mean())

# 7. Visualizations
sns.countplot(data=df, x="sex", hue="survived")
plt.title("Survival by Gender")
plt.show()

sns.countplot(data=df, x="pclass", hue="survived")
plt.title("Survival by Passenger Class")
plt.show()

sns.histplot(data=df, x="age", kde=True)
plt.title("Age Distribution")
plt.show()

sns.boxplot(data=df, x="pclass", y="fare")
plt.title("Fare by Passenger Class")
plt.show()

# 8. Correlation
numeric_columns = ["survived", "pclass", "age", "sibsp", "parch", "fare"]
correlation = df[numeric_columns].corr()

sns.heatmap(correlation, annot=True)
plt.title("Correlation Heatmap")
plt.show()