Module 5: File Handling & I /O

M5: File Handling & I/O — Python for Corporate Professionals | OTLMS
M5 · File Handling & I/O Python for Corporate Professionals

Reading, Writing, and Managing Files from Python

Every real-world script eventually needs to read from or write to the filesystem — log files, configuration files, CSV exports, JSON from an API, reports. This module bridges the gap between Python code that runs in memory and data that persists on disk. By the end, you’ll be able to read a CSV, process it, and write clean results back — all from a single script.

5 topics ~80 min read Intermediate level All roles
📖
Concept — How Python Talks to the Filesystem
Text files, CSV, JSON, paths, and config — the five pillars of file I/O

Everything you’ve written so far has lived and died with the program run. The moment your script finishes, all your variables disappear. Files break that cycle — they let your script read data that existed before it ran, and leave data behind for the next run, for a human to read, or for another system to consume.

Topic 1

Text Files — open(), read, write, and the with Statement

Opening a file in Python takes one line: open(filename, mode). The mode tells Python what you intend to do with the file. Getting the mode wrong is one of the most common beginner mistakes — “w” will silently erase everything in an existing file the moment you open it.

“r”
Read only. File must exist.
“w”
Write. Creates file if missing.
⚠ Erases existing content.
“a”
Append. Adds to end of file. Safe for logs.
“x”
Create new file. Fails if file already exists.
“r+”
Read and write. File must exist.
“rb” / “wb”
Binary mode. For images, PDFs, non-text files.

Always use the with statement when working with files. It guarantees that the file is properly closed when the block finishes — even if an error occurs. Forgetting to close a file can leave it locked or cause data corruption. The with statement handles this automatically.

with open(“servers.txt”, “r”, encoding=“utf-8”) as f:opens file, assigns to f content = f.read()file is open here — read/write inside # file is automatically closed hereclosed even if an error occurred above

Always pass encoding=”utf-8″ explicitly. On Windows, Python defaults to a system encoding that varies by locale and will cause silent bugs when files contain non-ASCII characters — anything with Indian language names, special symbols, or currency signs like ₹. UTF-8 is the universal standard and the right default.

Three ways to read a file: .read() returns the entire content as one string. .readlines() returns a list where each item is one line (including the trailing newline). Iterating directly over the file object (for line in f:) reads one line at a time without loading everything into memory — the right choice for large files.

Topic 2

CSV Files — Reading and Writing Tabular Data

CSV (Comma-Separated Values) is the most common format for exchanging tabular data — exports from Excel, database dumps, reports from monitoring systems. Python’s built-in csv module handles them correctly, including fields that contain commas or quotes inside them.

Never manually split on commas to parse a CSV. That works for trivial cases but breaks immediately on any field that contains a comma inside quotes. The csv module handles all of that for you.

There are two reading styles. csv.reader gives you each row as a plain list — you access fields by index (row[0], row[1]). csv.DictReader gives you each row as a dictionary with column headers as keys — you access fields by name (row[“server_name”]). DictReader is almost always the better choice: it makes your code self-documenting and survives column reordering.

DictReader vs reader: If someone changes the column order in the CSV, code using row[0] silently reads the wrong data. Code using row[“server_name”] continues to work correctly. Use DictReader unless you have a specific reason not to.
Topic 3

JSON Files — Structured Data In and Out

JSON (JavaScript Object Notation) is the lingua franca of APIs, config files, and structured data exchange. A JSON object maps directly to a Python dictionary. A JSON array maps to a Python list. Once loaded, you work with it as pure Python — no special methods needed.

The json module has four key functions. json.load(f) reads JSON from a file object. json.loads(string) parses a JSON string (the “s” stands for string — useful for API responses). json.dump(data, f) writes Python data as JSON to a file. json.dumps(data) converts Python data to a JSON string without writing anywhere.

FunctionDirectionInput/Output
json.load(f)JSON → PythonReads from a file object
json.loads(s)JSON string → PythonParses a string variable (e.g. API response)
json.dump(obj, f)Python → JSONWrites to a file object
json.dumps(obj)Python → JSON stringReturns a string, writes nothing

When writing JSON, pass indent=2 to json.dump() to produce human-readable output. Without it, the entire file is on one line — perfectly valid JSON but impossible for a human to read or edit.

Topic 4

Path Operations with os.path and pathlib

File paths look deceptively simple until your script runs on a different operating system and the slashes go the wrong way, or you try to join two path strings with + and produce “logs/app.log” on one machine and “logsapp.log” on another. Always build paths with os.path.join() or Python’s newer pathlib module — never with string concatenation.

os.path gives you a set of functions for path manipulation. The ones you’ll use most:

FunctionWhat it doesExample result
os.path.join(a, b)Joins path components with the correct separator“logs/app.log”
os.path.exists(p)True if file or directory existsTrue / False
os.path.isfile(p)True only if it’s a file (not a directory)True / False
os.path.isdir(p)True only if it’s a directoryTrue / False
os.path.basename(p)File name from a full path“app.log”
os.path.dirname(p)Directory portion of a path“/var/log”
os.path.splitext(p)Splits name and extension as a tuple(“/var/log/app”, “.log”)
os.makedirs(p, exist_ok=True)Creates directory and any missing parents(creates dirs)
💡 pathlib alternative: Python 3.4+ includes pathlib.Path, which lets you write Path(“logs”) / “app.log” using the / operator to join paths. It’s cleaner for complex path operations. Both os.path and pathlib are correct — use whichever your team already uses.
Topic 5

Config Files with configparser

Hard-coding database hostnames, passwords, and API URLs directly in your Python scripts is a maintenance problem — changing a server means editing source code. Config files solve this: keep the settings in a separate .ini file, and your script reads them at runtime.

Python’s configparser module reads .ini-style config files with sections and key-value pairs. This format is readable, editable without Python knowledge, and universally understood. A typical config file looks like this:

📄 config.ini
[database]
host     = ora-prod-01.internal
port     = 1521
name     = ORCL
user     = app_user

[logging]
level    = INFO
log_dir  = /var/log/myapp
max_days = 30

[api]
endpoint = https://api.internal.company.com
timeout  = 10

Access is dictionary-like: config[“database”][“host”]. Use config.get(“database”, “host”) for safe access with an optional fallback. All values are strings by default — use config.getint(), config.getfloat(), or config.getboolean() for typed reads.

⚠️ Never commit config files containing real passwords or API keys to version control. In production, read secrets from environment variables (os.environ.get(“DB_PASSWORD”)) or a secrets manager. The config file pattern shown here is for non-sensitive settings like hostnames, ports, and log levels.
✏️
Syntax Reference
Every file operation pattern you’ll use — text, CSV, JSON, paths, config

Text files — read and write

Python
# ── READING ──

# Read entire file as one string
with open("servers.txt", "r", encoding="utf-8") as f:
    content = f.read()                     # one big string

# Read as a list of lines
with open("servers.txt", "r", encoding="utf-8") as f:
    lines = f.readlines()                  # ["line1\n", "line2\n", ...]

# Read line by line (memory-efficient for large files)
with open("servers.txt", "r", encoding="utf-8") as f:
    for line in f:
        line = line.strip()                # removes \n and leading/trailing spaces
        if line and not line.startswith("#"):  # skip blank lines and comments
            print(line)

# ── WRITING ──

# Write (creates or overwrites)
with open("output.txt", "w", encoding="utf-8") as f:
    f.write("Server report\n")
    f.write("==============\n")

# Write multiple lines at once
lines_to_write = ["web-01\n", "db-01\n", "api-01\n"]
with open("servers.txt", "w", encoding="utf-8") as f:
    f.writelines(lines_to_write)

# Append — safe for logs (doesn't erase existing content)
with open("app.log", "a", encoding="utf-8") as f:
    f.write("[2026-06-15 09:41] Server started\n")

CSV files — read and write

Python
import csv

# ── READING with DictReader (recommended) ──
with open("servers.csv", "r", encoding="utf-8", newline="") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row["server_name"], row["cpu"])  # access by column name

# Load all rows into a list of dicts (for in-memory processing)
with open("servers.csv", "r", encoding="utf-8", newline="") as f:
    rows = list(csv.DictReader(f))       # list of dicts, one per row

# ── READING with reader (access by index) ──
with open("servers.csv", "r", encoding="utf-8", newline="") as f:
    reader = csv.reader(f)
    header = next(reader)               # skip the header row
    for row in reader:
        print(row[0], row[1])            # access by position

# ── WRITING with DictWriter ──
fieldnames = ["server_name", "environment", "cpu", "status"]
data = [
    {"server_name": "prod-web-01", "environment": "production", "cpu": 82, "status": "OK"},
    {"server_name": "dev-01",       "environment": "dev",        "cpu": 14, "status": "OK"},
]

with open("report.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()                   # writes the column header row
    writer.writerows(data)                 # writes all rows at once

# Note: always pass newline="" when opening CSV files in Python 3
# to prevent an extra blank line being written between rows on Windows

JSON files — read and write

Python
import json

# ── READING ──
with open("config.json", "r", encoding="utf-8") as f:
    data = json.load(f)                   # file → Python dict/list

# Parse a JSON string (e.g. from requests.get().text or an API response)
json_string = '{"server": "prod-01", "cpu": 82, "online": true}'
parsed = json.loads(json_string)          # string → Python dict
print(parsed["server"])                  # "prod-01"

# ── WRITING ──
servers = [
    {"name": "prod-web-01", "cpu": 82, "online": True},
    {"name": "dev-01",       "cpu": 14, "online": True},
]

with open("servers.json", "w", encoding="utf-8") as f:
    json.dump(servers, f, indent=2)       # indent=2 for human-readable output

# Convert to string without writing to file
json_str = json.dumps(servers, indent=2)

# Update an existing JSON file (read → modify → write back)
with open("servers.json", "r", encoding="utf-8") as f:
    data = json.load(f)

data.append({"name": "stg-01", "cpu": 30, "online": True})

with open("servers.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

Path operations with os.path

Python
import os

# Build paths safely (works on Windows, Linux, and macOS)
log_path   = os.path.join("var", "log", "app.log")  # "var/log/app.log"
report_dir = os.path.join(os.getcwd(), "reports")

# Check before acting
if os.path.exists(log_path):
    print("Log file found")
if os.path.isfile(log_path):
    print("It's a file, not a directory")

# Extract parts of a path
full_path = "/var/log/nginx/access.log"
os.path.basename(full_path)            # "access.log"
os.path.dirname(full_path)             # "/var/log/nginx"
name, ext = os.path.splitext(os.path.basename(full_path))
# name = "access"   ext = ".log"

# Create directories (including any missing parents)
os.makedirs(os.path.join("reports", "2026", "june"), exist_ok=True)
# exist_ok=True means no error if directory already exists

# List files in a directory
for filename in os.listdir("logs"):
    filepath = os.path.join("logs", filename)
    if os.path.isfile(filepath) and filename.endswith(".log"):
        size_kb = os.path.getsize(filepath) / 1024
        print(f"{filename}: {size_kb:.1f} KB")

Config files with configparser

Python
import configparser

config = configparser.ConfigParser()
config.read("config.ini", encoding="utf-8")

# Access values — all returned as strings by default
host = config["database"]["host"]              # "ora-prod-01.internal"
port = config.getint("database", "port")        # 1521 — returned as int
days = config.getint("logging", "max_days")     # 30
tmo  = config.getfloat("api", "timeout")        # 10.0

# Safe access with a fallback default
debug = config.getboolean("logging", "debug", fallback=False)
level = config.get("logging", "level", fallback="INFO")

# Check if a section or key exists
if "api" in config:
    endpoint = config["api"]["endpoint"]

# List all sections and their keys
for section in config.sections():
    print(f"[{section}]")
    for key, value in config[section].items():
        print(f"  {key} = {value}")

# Write a config file programmatically
new_config = configparser.ConfigParser()
new_config["database"] = {"host": "localhost", "port": "5432"}
new_config["logging"]  = {"level": "DEBUG"}

with open("dev_config.ini", "w", encoding="utf-8") as f:
    new_config.write(f)
💡
Examples — Files Doing Real Work
Five complete programs that read from, process, and write to files

Each example shows both the file content (what your script reads or writes) and the Python code that processes it. Study the pair together — the file structure explains the code, and the code explains the file structure.

Example 1 — IT Support: Log file parser
IT SupportDevOps

Reads a plain-text application log file line by line, counts errors and warnings, extracts the last 5 error lines, and writes a summary report. Uses efficient line-by-line reading so it works even on multi-gigabyte log files.

📄 app.log (input file)
[2026-06-14 08:12:01] INFO     Service started successfully
[2026-06-14 08:15:33] INFO     User login: priya.sharma@company.com
[2026-06-14 08:31:44] WARNING  Slow query detected: 4.2s (threshold: 3s)
[2026-06-14 09:02:17] ERROR    DB connection timeout: host=ora-prod-01 port=1521
[2026-06-14 09:02:45] ERROR    Retry 1/3 failed: ora-prod-01
[2026-06-14 09:03:11] ERROR    Retry 2/3 failed: ora-prod-01
[2026-06-14 09:03:38] INFO     Failover to ora-prod-02 successful
[2026-06-14 09:44:02] WARNING  Disk usage above 80%: /var/log (83%)
[2026-06-14 10:15:09] ERROR    Authentication failed: user=svc_backup
[2026-06-14 10:55:21] INFO     Scheduled backup completed
Python
import os
from datetime import datetime

def parse_log(log_path):
    """Read a log file and return counts and recent errors."""
    counts    = {"INFO": 0, "WARNING": 0, "ERROR": 0}
    error_lines = []

    with open(log_path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            for level in counts:
                if level in line:
                    counts[level] += 1
                    if level == "ERROR":
                        error_lines.append(line)
                    break

    return counts, error_lines[-5:]    # last 5 errors

def write_summary(log_path, counts, recent_errors, out_path):
    """Write a plain-text summary report."""
    ts        = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    log_name  = os.path.basename(log_path)
    total     = sum(counts.values())

    with open(out_path, "w", encoding="utf-8") as f:
        f.write(f"Log Analysis Report\n")
        f.write(f"Generated : {ts}\n")
        f.write(f"Source    : {log_name}\n")
        f.write(f"{'─' * 40}\n\n")
        f.write(f"Total lines : {total}\n")
        for level, count in counts.items():
            pct = (count / total * 100) if total else 0
            f.write(f"  {level:8}: {count:>4} ({pct:.1f}%)\n")

        if recent_errors:
            f.write(f"\nRecent errors ({len(recent_errors)}):\n")
            for err in recent_errors:
                f.write(f"  {err}\n")

# ── Run ──
log_file = "app.log"
out_file = "log_summary.txt"
counts, recent_errors = parse_log(log_file)
write_summary(log_file, counts, recent_errors, out_file)

print(f"Summary written to {out_file}")
print(f"Errors: {counts['ERROR']}  Warnings: {counts['WARNING']}")
Output (terminal)
Summary written to log_summary.txt
Errors: 4  Warnings: 2
📄 log_summary.txt (written by script)
Log Analysis Report
Generated : 2026-06-15 09:41:05
Source    : app.log
────────────────────────────────────────

Total lines : 10
  INFO    :    5 (50.0%)
  WARNING :    2 (20.0%)
  ERROR   :    3 (30.0%)

Recent errors (3):
  [2026-06-14 09:02:17] ERROR    DB connection timeout: host=ora-prod-01 port=1521
  [2026-06-14 09:03:11] ERROR    Retry 2/3 failed: ora-prod-01
  [2026-06-14 10:15:09] ERROR    Authentication failed: user=svc_backup
Example 2 — Database Developer: CSV to filtered report
Database DevAutomation

Reads an employee CSV exported from an HR database, filters for active employees earning above a threshold, calculates department totals, and writes a clean filtered CSV. Demonstrates the full CSV read-process-write workflow.

📄 employees.csv (input)
emp_id,name,department,salary,active
E001,Priya Sharma,IT,85000,true
E002,Rahul Mehta,IT,92000,true
E003,Anita Patel,Finance,78000,true
E004,Suresh Kumar,IT,67000,false
E005,Meera Nair,Finance,95000,true
E006,Kiran Desai,HR,55000,true
E007,Arjun Singh,IT,110000,true
Python
import csv

def load_employees(filepath):
    """Load CSV into a list of dicts with type conversion."""
    employees = []
    with open(filepath, "r", encoding="utf-8", newline="") as f:
        for row in csv.DictReader(f):
            employees.append({
                "emp_id":     row["emp_id"],
                "name":       row["name"],
                "department": row["department"],
                "salary":     int(row["salary"]),
                "active":     row["active"].lower() == "true",
            })
    return employees

def filter_active_above(employees, min_salary):
    """Return active employees earning at or above min_salary."""
    return [e for e in employees
            if e["active"] and e["salary"] >= min_salary]

def dept_totals(employees):
    """Return dict of department → total salary."""
    totals = {}
    for e in employees:
        totals[e["department"]] = totals.get(e["department"], 0) + e["salary"]
    return totals

def write_report_csv(employees, filepath):
    """Write filtered employees to a new CSV file."""
    fields = ["emp_id", "name", "department", "salary"]
    with open(filepath, "w", encoding="utf-8", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
        writer.writeheader()
        writer.writerows(employees)

# ── Run ──
all_emp   = load_employees("employees.csv")
filtered  = filter_active_above(all_emp, min_salary=75000)
totals    = dept_totals(filtered)

write_report_csv(filtered, "high_earners.csv")

print(f"Filtered {len(filtered)} of {len(all_emp)} employees → high_earners.csv")
print("\nDepartment payroll (active, ≥₹75,000):")
for dept, total in sorted(totals.items()):
    print(f"  {dept:12}: ₹{total:,}")
Output
Filtered 4 of 7 employees → high_earners.csv

Department payroll (active, ≥₹75,000):
  Finance      : ₹1,73,000
  IT           : ₹2,87,000
Example 3 — DevOps: JSON config manager
DevOps

Reads a deployment configuration from a JSON file, validates required fields, adds a deployment timestamp, and writes the updated config back. Shows the full JSON read-modify-write cycle with safe path handling using os.path.

📄 deployment.json (input)
{
  "app_name": "customer-portal",
  "version": "2.4.1",
  "environments": {
    "production": {
      "replicas": 3,
      "db_host": "ora-prod-01.internal",
      "ssl": true
    },
    "staging": {
      "replicas": 1,
      "db_host": "ora-stg-01.internal",
      "ssl": true
    }
  },
  "last_deployed": null
}
Python
import json
import os
from datetime import datetime

REQUIRED_FIELDS = ["app_name", "version", "environments"]

def load_deployment_config(path):
    """Load and validate a JSON deployment config. Returns dict or raises."""
    if not os.path.isfile(path):
        raise FileNotFoundError(f"Config not found: {path}")

    with open(path, "r", encoding="utf-8") as f:
        config = json.load(f)

    missing = [k for k in REQUIRED_FIELDS if k not in config]
    if missing:
        raise ValueError(f"Config missing required fields: {missing}")

    return config

def deploy(config, target_env):
    """Simulate deployment: update config with timestamp and env info."""
    if target_env not in config["environments"]:
        raise KeyError(f"Unknown environment: {target_env}")

    env_cfg = config["environments"][target_env]
    config["last_deployed"] = {
        "environment": target_env,
        "timestamp":   datetime.now().strftime("%Y-%m-%dT%H:%M:%S"),
        "replicas":    env_cfg["replicas"],
        "db_host":     env_cfg["db_host"],
    }
    return config

def save_config(config, path):
    """Write config back to disk as indented JSON."""
    with open(path, "w", encoding="utf-8") as f:
        json.dump(config, f, indent=2)

# ── Run ──
config_path = "deployment.json"
target      = "staging"

config = load_deployment_config(config_path)
print(f"Loaded config for {config['app_name']} v{config['version']}")

config = deploy(config, target)
save_config(config, config_path)

dep = config["last_deployed"]
print(f"Deployed to {dep['environment']} at {dep['timestamp']}")
print(f"DB host  : {dep['db_host']}")
print(f"Replicas : {dep['replicas']}")
print(f"Config saved → {os.path.abspath(config_path)}")
Output
Loaded config for customer-portal v2.4.1
Deployed to staging at 2026-06-15T09:41:05
DB host  : ora-stg-01.internal
Replicas : 1
Config saved → /home/devops/scripts/deployment.json
Example 4 — AI / ML: Dataset manifest builder
AI / ML

Scans a directory of CSV training data files, reads the header row of each, counts rows, and writes a JSON manifest describing the dataset. Uses os.listdir(), os.path, csv, and json together — a realistic data-pipeline housekeeping task.

Python
import os
import csv
import json
from datetime import datetime

def inspect_csv(filepath):
    """Return column names and row count for a CSV file."""
    with open(filepath, "r", encoding="utf-8", newline="") as f:
        reader = csv.DictReader(f)
        columns = reader.fieldnames or []
        row_count = sum(1 for _ in reader)
    return list(columns), row_count

def build_manifest(data_dir, output_path):
    """
    Scan data_dir for CSV files and write a JSON manifest.

    Args:
        data_dir    (str): Directory containing CSV training files.
        output_path (str): Where to write the manifest JSON.
    """
    if not os.path.isdir(data_dir):
        raise NotADirectoryError(f"Not a directory: {data_dir}")

    datasets = []
    for filename in sorted(os.listdir(data_dir)):
        if not filename.endswith(".csv"):
            continue
        filepath  = os.path.join(data_dir, filename)
        size_kb   = os.path.getsize(filepath) / 1024
        cols, rows = inspect_csv(filepath)
        name, _   = os.path.splitext(filename)

        datasets.append({
            "name":        name,
            "filename":    filename,
            "rows":        rows,
            "columns":     cols,
            "column_count": len(cols),
            "size_kb":     round(size_kb, 2),
        })

    manifest = {
        "generated_at": datetime.now().strftime("%Y-%m-%dT%H:%M:%S"),
        "data_dir":     os.path.abspath(data_dir),
        "total_files":  len(datasets),
        "total_rows":   sum(d["rows"] for d in datasets),
        "datasets":     datasets,
    }

    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(manifest, f, indent=2)

    return manifest

# ── Run ──
manifest = build_manifest("training_data", "dataset_manifest.json")

print(f"Manifest built: {manifest['total_files']} files, "
      f"{manifest['total_rows']} total rows")
print()
for ds in manifest["datasets"]:
    print(f"  {ds['name']:25} {ds['rows']:>6} rows  "
          f"{ds['column_count']} cols  {ds['size_kb']} KB")
Output
Manifest built: 3 files, 14820 total rows

  support_tickets            8432 rows  9 cols  412.5 KB
  billing_events             5100 rows  7 cols  198.2 KB
  user_sessions              1288 rows  12 cols  88.7 KB
Example 5 — Automation: Config-driven script
AutomationDevOpsAll Roles

A script that reads all its runtime settings from a .ini config file — database connection, log location, report directory — and generates a daily report based on those settings. Shows the correct pattern for production scripts that should never require code edits to change their behaviour.

📄 monitor.ini
[database]
host     = ora-prod-01.internal
port     = 1521
name     = PRODDB

[reporting]
output_dir   = reports
filename_fmt = server_report_%Y%m%d.txt
alert_cpu    = 80
alert_mem    = 85

[logging]
log_file = monitor.log
level    = INFO
Python
import configparser
import os
from datetime import datetime

# Simulated server data (in a real script this comes from an API or SSH)
FLEET = [
    {"name": "prod-web-01", "cpu": 88, "mem": 72, "disk": 55},
    {"name": "prod-db-01",  "cpu": 62, "mem": 91, "disk": 83},
    {"name": "prod-api-01", "cpu": 44, "mem": 60, "disk": 40},
]

def load_config(ini_path):
    """Load and return the config. Raises if file is missing."""
    if not os.path.isfile(ini_path):
        raise FileNotFoundError(f"Config file not found: {ini_path}")
    cfg = configparser.ConfigParser()
    cfg.read(ini_path, encoding="utf-8")
    return cfg

def check_alerts(servers, cpu_thresh, mem_thresh):
    """Return list of alert strings for servers breaching thresholds."""
    alerts = []
    for s in servers:
        if s["cpu"] > cpu_thresh:
            alerts.append(f"CPU ALERT  {s['name']}: {s['cpu']}% (threshold {cpu_thresh}%)")
        if s["mem"] > mem_thresh:
            alerts.append(f"MEM ALERT  {s['name']}: {s['mem']}% (threshold {mem_thresh}%)")
    return alerts

def write_report(cfg, servers, alerts):
    """Write the daily report to the configured output directory."""
    out_dir    = cfg.get("reporting", "output_dir")
    fmt        = cfg.get("reporting", "filename_fmt")
    filename   = datetime.now().strftime(fmt)
    report_path = os.path.join(out_dir, filename)

    os.makedirs(out_dir, exist_ok=True)

    with open(report_path, "w", encoding="utf-8") as f:
        f.write(f"Daily Server Report — {datetime.now().strftime('%d %b %Y %H:%M')}\n")
        f.write(f"Database : {cfg['database']['host']} / {cfg['database']['name']}\n")
        f.write(f"{'─' * 50}\n\n")

        f.write(f"{'Server':18} {'CPU':>5} {'Mem':>5} {'Disk':>6}\n")
        f.write(f"{'─' * 40}\n")
        for s in servers:
            f.write(f"{s['name']:18} {s['cpu']:>4}% {s['mem']:>4}% {s['disk']:>5}%\n")

        f.write(f"\nAlerts ({len(alerts)}):\n")
        if alerts:
            for a in alerts:
                f.write(f"  ⚠ {a}\n")
        else:
            f.write("  No alerts.\n")

    return report_path

# ── Run ──
cfg     = load_config("monitor.ini")
alerts  = check_alerts(
    FLEET,
    cpu_thresh = cfg.getint("reporting", "alert_cpu"),
    mem_thresh = cfg.getint("reporting", "alert_mem")
)
path = write_report(cfg, FLEET, alerts)
print(f"Report written: {path}")
print(f"Alerts: {len(alerts)}")
for a in alerts:
    print(f"  {a}")
Output
Report written: reports/server_report_20260615.txt
Alerts: 2
  CPU ALERT  prod-web-01: 88% (threshold 80%)
  MEM ALERT  prod-db-01: 91% (threshold 85%)
🏋️
Practice Exercises
Four exercises — one per file type, each building on what came before

Create the sample input files yourself before running your code — this is realistic practice. In production you’ll always be working with files you didn’t create. The act of writing the sample file by hand also helps you understand the format your code needs to parse.

1
Text file word counter. Create a plain text file called incident_report.txt with at least 8 lines describing an IT incident (any realistic content). Write a Python script that reads the file and prints: total number of lines, total number of words, total number of characters (excluding newlines), and the 5 most common words (case-insensitive, ignoring words shorter than 4 characters). Use a dictionary to count word frequencies.
Read all lines with a for line in f loop. Split each line into words with line.lower().split(). For word counting: counts[word] = counts.get(word, 0) + 1. To find the top 5: sorted(counts.items(), key=lambda x: x[1], reverse=True)[:5]. Filter short words: if len(word) >= 4 before counting.
2
CSV salary band classifier. Create a CSV file called staff.csv with columns: name, role, salary (at least 6 rows). Write a script that reads it with DictReader, classifies each employee into a salary band (Junior: below ₹60,000 / Mid: ₹60,000–₹90,000 / Senior: above ₹90,000), and writes a new CSV called staff_banded.csv with all original columns plus a new band column. Remember to convert salary to an integer before comparing.
When reading: salary = int(row[“salary”]). Write a classify_band(salary) function that returns the band string. When writing, add “band” to the fieldnames list for DictWriter. Before calling writer.writerows(), make sure each row dict has a “band” key: row[“band”] = classify_band(int(row[“salary”])).
3
JSON to-do list manager. Write a script that manages a persistent to-do list stored in a JSON file called todos.json. It should have three functions: load_todos() that returns an empty list if the file doesn’t exist yet, add_todo(text) that appends a new item with {“text”: text, “done”: false, “added”: timestamp} and saves, and mark_done(index) that sets done: true on the item at that index and saves. Call all three in a demo: add 3 items, mark one done, then print the list.
For load_todos(): use if not os.path.exists(“todos.json”): return [] before trying to open. For saving, wrap the json.dump() call in its own save_todos(todos) function so both add_todo and mark_done can call it. For the timestamp: datetime.now().strftime(“%Y-%m-%d %H:%M”).
4
Config-driven directory scan. Create a scan.ini file with two sections: [scan] with keys directory and extension (e.g. .py or .txt), and [output] with key report_file. Write a Python script that reads this config, scans the specified directory for files with the given extension, and writes a report (file name, size in KB, modification date) to the configured output file. The script should work correctly if you change scan.ini without touching the Python code.
Read the config: cfg = configparser.ConfigParser(); cfg.read(“scan.ini”). Get directory: cfg[“scan”][“directory”]. Scan with os.listdir(), filter with filename.endswith(ext). For modification date: datetime.fromtimestamp(os.path.getmtime(filepath)).strftime(“%Y-%m-%d”). Write the report with open(report_file, “w”).
📋
Assignment — M5
A complete data processing pipeline — estimated 50–65 minutes

📋 Employee Payroll Pipeline

Build a complete data processing pipeline that reads raw data from files, processes it, and produces multiple output files. The pipeline must be driven by a .ini config file — no paths, thresholds, or settings should be hard-coded in the Python script.

  1. Create payroll.ini with sections for [input] (CSV file path), [output] (directory, summary JSON filename, flagged CSV filename), and [rules] (salary thresholds for Junior/Mid/Senior bands, a minimum salary below which an employee is flagged for review).
  2. Create employees.csv with at least 10 rows and columns: emp_id, name, department, salary, active, join_year. Include a mix of departments, salary ranges, and active/inactive status.
  3. Write payroll_pipeline.py with these functions (all settings read from config):
    • load_config(path) — reads and validates the INI file
    • load_employees(path) — reads CSV, converts types (salary to int, active to bool, join_year to int)
    • classify_band(salary, cfg) — returns band string using thresholds from config
    • process(employees, cfg) — adds band and flagged fields to each active employee
    • write_summary_json(employees, path) — writes a JSON file with: total employees, active count, department breakdown (count + total salary per dept), band distribution, and list of flagged employees
    • write_flagged_csv(employees, path) — writes only flagged employees to a CSV
  4. Output: the script should print a brief run summary to the terminal (files written, counts, flagged count) and create the two output files in the configured output directory.
Your pipeline is complete when changing only payroll.ini — different directory, different thresholds, different input file — causes the script to behave correctly without any code changes. Paste your INI file, a sample of your CSV, and the terminal output in the comments.
🧠
Quiz — Check Your Understanding
5 questions · instant feedback · retake as many times as you need

These questions test both syntax knowledge and judgment — knowing which tool to reach for and why. A few are deliberately tricky: the wrong answer is tempting but causes a real bug.

1. You open an existing log file with open(“app.log”, “w”) and write a new entry. What happens to the previous contents of the file?
2. You receive a CSV where the columns are: server_id, hostname, cpu_pct, status. A colleague changes the export so the columns are now in the order: hostname, server_id, status, cpu_pct. Which reading approach survives this change without breaking?
3. You call json.load(f) on a JSON file that contains {“port”: 1521, “ssl”: true, “timeout”: null}. What Python types will port, ssl, and timeout have after loading?
4. What is the correct way to build a file path that works on both Windows and Linux?
5. You use config.getint(“database”, “port”) to read a port number from a config file, but the key “port” is missing from that section. What happens?
Your scripts can now read, process, and write real files. One gap remains: what happens when something goes wrong?
M6: Error Handling — try/except, custom exceptions, and writing scripts that fail gracefully.
Start M6 →

Scroll to Top