Python Pandas

pandas is the most popular Python library for working with tables of data: sales records, survey answers, exam results, anything you might keep in a spreadsheet. Its main type is the DataFrame, a table with named columns, and a Series, a single column. pandas reads CSV and Excel files, cleans and filters data, and summarises it in a few lines. It is installed in our editor; the first import pandas takes a moment to set up.

A first DataFrame

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Ravi", "Meera", "John"],
    "city": ["Pune", "Delhi", "Pune", "Goa"],
    "marks": [91, 78, 85, 64],
})
print(df)
print(df.shape)
print(df["marks"].mean())

    name   city  marks
0   Asha   Pune     91
1   Ravi  Delhi     78
2  Meera   Pune     85
3   John    Goa     64
(4, 3)
79.5

Each key of the dictionary becomes a column. The numbers on the left are the index, the labels of the rows. df["marks"] selects one column as a Series, which has methods such as mean, max and sum.

Filtering, sorting and new columns

import pandas as pd

df = pd.DataFrame({"name": ["Asha", "Ravi", "Meera", "John"], "marks": [91, 78, 85, 64]})
print(df[df["marks"] >= 80])
print(df.sort_values("marks", ascending=False)["name"].tolist())
df["result"] = df["marks"].apply(lambda m: "pass" if m >= 70 else "fail")
print(df)

    name  marks
0   Asha     91
2  Meera     85
['Asha', 'Meera', 'Ravi', 'John']
    name  marks result
0   Asha     91   pass
1   Ravi     78   pass
2  Meera     85   pass
3   John     64   fail

A condition on a column gives a mask, just as in NumPy, and df[mask] keeps the matching rows. Assigning to a new column name adds a column.

Reading a CSV file and grouping

Real data usually comes from a file. pd.read_csv reads a CSV file into a DataFrame, and groupby summarises the rows by the values of a column:

import pandas as pd

with open("sales.csv", "w") as f:
    f.write("city,month,amount\nPune,Jan,120\nDelhi,Jan,200\nPune,Feb,150\nDelhi,Feb,180\nGoa,Feb,90\n")

sales = pd.read_csv("sales.csv")
print(sales.groupby("city")["amount"].sum())
print(sales.groupby("month")["amount"].agg(["sum", "mean"]))

city
Delhi    380
Goa       90
Pune     270
Name: amount, dtype: int64
       sum   mean
month            
Feb    420  140.0
Jan    320  160.0

agg applies several summaries at once. df.to_csv("result.csv", index=False) saves a DataFrame back to a file, which then appears in the file list.

Looking at data

df.head() shows the first rows, df.info() the columns and their types, and df.describe() the count, mean, minimum, maximum and quartiles of every number column: three quick ways to get to know a new dataset.

Practice exercises

1. Make a DataFrame of five products with their prices and quantities, and add a column for the total value.
2. Print only the products that cost more than 100, sorted by price.
3. Write a CSV file of students and cities, read it with read_csv, and count the students per city with groupby and size().
4. Print describe() for a column of numbers and explain each line.