KAHIBARO Bookshelves
Discord Login Register

16.4. Introduction to pandas

What is pandas?

In data science, you often work with tables of data: rows and columns, like a spreadsheet.
pandas is a Python library that makes working with this kind of data much easier.

pandas gives you two main tools:

  • Series: a one-dimensional labeled array (like a column in a table)
  • DataFrame: a two-dimensional labeled table (like a whole spreadsheet)

In this chapter you will:

  • Create simple pandas objects
  • Load data from a CSV file
  • View and explore data
  • Select rows and columns
  • Do basic calculations and filtering

You don’t need to memorize everything—this chapter is about getting a feel for how pandas works.

Installing and importing pandas

pandas is not part of Python’s standard library, so you usually install it with pip (covered earlier):

pip install pandas

In your Python code, you almost always import it like this:

import pandas as pd

The as pd is a common shortcut, so you can write pd.DataFrame() instead of pandas.DataFrame().

The Series: a labeled one-dimensional array

A Series is like a single column with labels (called index).

Creating a Series

You can create a Series from a Python list:

import pandas as pd
numbers = [10, 20, 30]
s = pd.Series(numbers)
print(s)

Output (the left side is the index, the right side is the value):

0    10
1    20
2    30
dtype: int64
  • The numbers 0, 1, 2 on the left are the index labels.
  • dtype shows the data type.

Custom index labels

You can give your own labels:

temperatures = [18.5, 20.0, 21.2]
days = ["Mon", "Tue", "Wed"]
s = pd.Series(temperatures, index=days)
print(s)

Output:

Mon    18.5
Tue    20.0
Wed    21.2
dtype: float64

You can access a value by its label:

print(s["Tue"])   # 20.0

The DataFrame: working with tables

A DataFrame is the main structure you will use in pandas.
It looks and behaves like a table:

  • Rows (with an index)
  • Columns (with names)
  • Each column can have its own data type

Creating a DataFrame from a dictionary

A very common way to build a DataFrame in code is from a dictionary of lists:

import pandas as pd
data = {
    "name": ["Alice", "Bob", "Charlie"],
    "age":  [25, 30, 35],
    "city": ["London", "Paris", "Berlin"]
}
df = pd.DataFrame(data)
print(df)

Output:

      name  age    city
0    Alice   25  London
1      Bob   30   Paris
2  Charlie   35  Berlin
  • Columns: name, age, city
  • Index: 0, 1, 2 (automatically created)

Custom index for rows

You can set your own row labels:

df = pd.DataFrame(data, index=["a", "b", "c"])
print(df)

Output:

      name  age    city
a    Alice   25  London
b      Bob   30   Paris
c  Charlie   35  Berlin

You can now use "a", "b", "c" to refer to rows.

Loading data from CSV files

In data science, you often receive data in CSV (Comma-Separated Values) files.

A simple CSV file (people.csv) might look like this:

name,age,city
Alice,25,London
Bob,30,Paris
Charlie,35,Berlin

Reading a CSV with pandas

Use read_csv:

import pandas as pd
df = pd.read_csv("people.csv")
print(df)

pandas will:

  • Read the file
  • Use the first row as column names
  • Create a DataFrame for you

If the file is not in the same folder, you will have to use a full or relative file path (covered earlier in the course).

Common `read_csv` options (beginner-friendly)

You don’t need many options to start, but these can be helpful:

  • sep – change the separator if it’s not a comma. For example: sep=";".
  • header – set which row is the header (column names). For example: header=0.

Examples:

df = pd.read_csv("data_semicolon.csv", sep=";")
df = pd.read_csv("data_no_header.csv", header=None)

If you set header=None, pandas will create default column names: 0, 1, 2, ...

Quick ways to inspect your data

Once you have a DataFrame, you’ll usually want to quickly look at it.

Assume this DataFrame:

import pandas as pd
data = {
    "name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
    "age":  [25, 30, 35, 40, 45],
    "city": ["London", "Paris", "Berlin", "London", "Paris"],
    "salary": [40000, 50000, 60000, 70000, 80000]
}
df = pd.DataFrame(data)

Viewing the first and last rows

  • head() shows the first few rows (default: 5):
print(df.head())
print(df.head(3))  # first 3 rows
  • tail() shows the last few rows:
print(df.tail())
print(df.tail(2))  # last 2 rows

Basic information about the DataFrame

  • df.shape shows number of rows and columns:
print(df.shape)  # (5, 4) -> 5 rows, 4 columns
  • df.columns shows column names:
print(df.columns)
# Index(['name', 'age', 'city', 'salary'], dtype='object')
  • df.info() gives a summary:
df.info()

It shows:

  • Number of rows
  • Column names and data types
  • How many non-missing values each column has

Quick statistics for numeric columns

Use describe() to get statistics:

print(df.describe())

Output (similar to):

             age        salary
count   5.000000      5.000000
mean   35.000000  60000.000000
std     7.905694  15811.388301
min    25.000000  40000.000000
25%    30.000000  50000.000000
50%    35.000000  60000.000000
75%    40.000000  70000.000000
max    45.000000  80000.000000

You don’t need to fully understand all of these yet, but mean, min, max are especially useful.

Selecting columns

Think of each column as a Series inside the DataFrame.

Selecting a single column

Use square brackets with the column name:

ages = df["age"]
print(ages)

Output:

0    25
1    30
2    35
3    40
4    45
Name: age, dtype: int64

You can also access df.age if the column name has no spaces or special characters:

print(df.age)

(Be aware this style can sometimes cause confusion with other attributes, so df["age"] is safer.)

Selecting multiple columns

Pass a list of column names inside the brackets:

name_and_city = df[["name", "city"]]
print(name_and_city)

Output:

      name    city
0    Alice  London
1      Bob   Paris
2  Charlie  Berlin
3    Diana  London
4      Eve   Paris

Selecting rows

There are several ways to select rows. Here are simple ones to start.

Selecting by row index position: `iloc`

iloc is used for selection by position (like list indexing):

# First row (position 0)
print(df.iloc[0])
# First three rows (positions 0, 1, 2)
print(df.iloc[0:3])

Selecting by row label: `loc`

loc uses labels. This is more interesting when you set a custom index.
Here, we will use the default numeric index as labels:

# Row with label 2 (third row)
print(df.loc[2])
# Rows with labels 1 to 3 (inclusive!)
print(df.loc[1:3])

Note the difference:

  • iloc[1:3] uses positions (like normal Python slicing: stops before 3)
  • loc[1:3] uses labels and includes both 1 and 3

Filtering rows with conditions

Filtering lets you select only rows that match some condition.

Simple condition on one column

For example, rows where age is greater than 30:

older_than_30 = df[df["age"] > 30]
print(older_than_30)

Output:

      name  age    city  salary
2  Charlie   35  Berlin   60000
3    Diana   40  London   70000
4      Eve   45   Paris   80000

What happens here:

  • df["age"] > 30 creates a Series of True/False values
  • df[ ... ] keeps only rows where the condition is True

Combining conditions

Use bitwise operators & (and) and | (or) with parentheses:

  • People older than 30 AND living in London:
mask = (df["age"] > 30) & (df["city"] == "London")
result = df[mask]
print(result)

Output:

    name  age    city  salary
3  Diana   40  London   70000
  • People in Paris OR with a salary over 75,000:
mask = (df["city"] == "Paris") | (df["salary"] > 75000)
result = df[mask]
print(result)

Adding and modifying columns

You can create new columns or update existing ones using simple expressions.

Adding a new column

Example: convert salary to thousands:

df["salary_k"] = df["salary"] / 1000
print(df)

Output:

      name  age    city  salary  salary_k
0    Alice   25  London   40000      40.0
1      Bob   30   Paris   50000      50.0
2  Charlie   35  Berlin   60000      60.0
3    Diana   40  London   70000      70.0
4      Eve   45   Paris   80000      80.0

Modifying an existing column

For example, increase all salaries by 10%:

df["salary"] = df["salary"] * 1.10
print(df[["name", "salary"]])

This changes the values in the salary column.

Handling missing data (basics)

Real-world data often has missing values.

pandas usually represents missing values as NaN (“Not a Number”).

Example:

data = {
    "name": ["Alice", "Bob", "Charlie"],
    "age":  [25, None, 35],
    "city": ["London", "Paris", None]
}
df = pd.DataFrame(data)
print(df)

Output:

      name   age    city
0    Alice  25.0  London
1      Bob   NaN   Paris
2  Charlie  35.0    NaN

Finding missing values

Use isna() (or isnull(), which is the same):

print(df.isna())

This shows True where the value is missing, False otherwise.

To count missing values in each column:

print(df.isna().sum())

Simple ways to deal with missing values

Two very common options:

  1. Drop rows with missing values:
   df_clean = df.dropna()
   print(df_clean)
  1. Fill missing values with some value (e.g., 0 or "Unknown"):
   df_filled = df.fillna({
       "age": 0,
       "city": "Unknown"
   })
   print(df_filled)

More advanced handling of missing data is usually part of deeper data cleaning, but this gives you a first taste.

Grouping and simple aggregation

pandas can quickly answer questions like:

  • What is the average salary per city?
  • How many people live in each city?

This is done with groupby and aggregation methods.

Assume this DataFrame:

data = {
    "city":   ["London", "Paris", "London", "Berlin", "Paris"],
    "salary": [40000, 50000, 45000, 55000, 60000]
}
df = pd.DataFrame(data)

Average salary per city

avg_salary_per_city = df.groupby("city")["salary"].mean()
print(avg_salary_per_city)

Output:

city
Berlin    55000.0
London    42500.0
Paris     55000.0
Name: salary, dtype: float64

Explanation:

  • groupby("city") groups the rows by city
  • ["salary"] focuses on the salary column
  • mean() calculates the average salary for each city

Counting rows per group

How many entries per city?

count_per_city = df["city"].value_counts()
print(count_per_city)

Output:

Paris     2
London    2
Berlin    1
Name: city, dtype: int64

value_counts() is a quick way to count how many times each value appears in a column.

Saving data to CSV

After working with your DataFrame, you might want to save the result to a CSV file.

Use to_csv:

df.to_csv("output.csv", index=False)
  • "output.csv" is the name of the file to create
  • index=False tells pandas not to write the row index to the file (usually what you want for simple exports)

If you omit index=False, the index will be saved as an extra column in the CSV.

Putting it all together: a tiny pandas workflow

Here is a short example that combines several of the ideas you’ve seen:

  1. Read data from a CSV
  2. Inspect it
  3. Filter rows
  4. Add a new column
  5. Compute a grouped statistic
  6. Save the result
import pandas as pd
# 1. Read data
df = pd.read_csv("employees.csv")
# 2. Quick look
print(df.head())
print(df.info())
# 3. Filter: keep only employees older than 30
df_filtered = df[df["age"] > 30]
# 4. Add a new column: yearly_bonus = 5% of salary
df_filtered["yearly_bonus"] = df_filtered["salary"] * 0.05
# 5. Average salary per department
avg_salary_per_dept = df_filtered.groupby("department")["salary"].mean()
print(avg_salary_per_dept)
# 6. Save filtered data to a new CSV
df_filtered.to_csv("employees_over_30.csv", index=False)

You don’t need to fully understand every line yet. The goal is to see how pandas lets you describe data tasks with relatively simple, readable code.

Tips for beginners using pandas

  • Always start with df.head() and df.info() to understand what your data looks like.
  • Use clear, descriptive column names when possible.
  • Don’t be afraid to print intermediate results (print(...)) while you’re learning.
  • Keep your experiments small: work with a few columns or rows until you understand what your code does.
  • Use the official documentation and simple search queries like “pandas filter rows” or “pandas groupby mean” to find examples.

As you continue with data science, you’ll use pandas constantly. This chapter gives you the basics so you can start exploring real datasets and simple analyses.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Chapter 1: What Is Programming?
    • ☰ 1.1. What is a computer program?
      • ☰ 1.2. What is Python?
        • ☰ 1.3. Where is Python used?
          • ☰ 1.4. How programming helps solve problems
            • ☰ 1.5. Your first Python program
            • ☰ 2. Chapter 2: Setting Up Python
              • ☰ 2.1. Installing Python (Windows / macOS / Linux)
                • ☰ 2.2. Using Python in the terminal
                  • ☰ 2.3. Using an IDE (IDLE / VS Code)
                    • ☰ 2.4. Running Python scripts
                      • ☰ 2.5. Python interactive mode
                      • ☰ 3. Chapter 3: Variables and Data Types
                        • ☰ 3.1. What is a variable?
                          • ☰ 3.2. Creating variables
                            • ☰ 3.3. Numbers
                              • ☰ 3.4. Test
                                • ☰ 3.5. Boolean values
                                  • ☰ 3.6. Type checking
                                  • ☰ 4. Chapter 4: Basic Operations
                                    • ☰ 4.1. Arithmetic operations
                                      • ☰ 4.2. Working with numbers
                                        • ☰ 4.3. String operations
                                          • ☰ 4.4. Combining text and variables
                                            • ☰ 4.5. Comments in Python
                                            • ☰ 5. Chapter 5: User Input and Output
                                              • ☰ 5.1. Getting input from the user
                                                • ☰ 5.2. Converting input to numbers
                                                  • ☰ 5.3. Formatting output
                                                    • ☰ 5.4. Writing interactive programs
                                                    • ☰ 6. Chapter 6: Conditions
                                                      • ☰ 6.1. Comparison operators
                                                        • ☰ 6.2. Logical operators
                                                          • ☰ 6.3. if, elif, else
                                                            • ☰ 6.4. Nested conditions
                                                              • ☰ 6.5. Common beginner mistakes
                                                              • ☰ 7. Chapter 7: Loops
                                                                • ☰ 7.1. Why loops are useful
                                                                  • ☰ 7.2. for loops
                                                                    • ☰ 7.3. while loops
                                                                      • ☰ 7.4. Loop counters
                                                                        • ☰ 7.5. break and continue
                                                                          • ☰ 7.6. Loop exercises
                                                                          • ☰ 8. Chapter 8: Data Collections
                                                                            • ☰ 8.1. Lists
                                                                              • ☰ 8.2. Tuples
                                                                                • ☰ 8.3. Dictionaries
                                                                                  • ☰ 8.4. Sets (basics)
                                                                                    • ☰ 8.5. Choosing the right data structure
                                                                                    • ☰ 9. Chapter 9: Functions
                                                                                      • ☰ 9.1. Why functions matter
                                                                                        • ☰ 9.2. Defining functions
                                                                                          • ☰ 9.3. Parameters and arguments
                                                                                            • ☰ 9.4. Return values
                                                                                              • ☰ 9.5. Scope (local vs global)
                                                                                                • ☰ 9.6. Reusing code
                                                                                                • ☰ 10. Chapter 10: Working with Files
                                                                                                  • ☰ 10.1. What is a file?
                                                                                                    • ☰ 10.2. Reading text files
                                                                                                      • ☰ 10.3. Writing and appending files
                                                                                                        • ☰ 10.4. File paths
                                                                                                          • ☰ 10.5. Simple file-based programs
                                                                                                          • ☰ 11. Chapter 11: Errors and Debugging
                                                                                                            • ☰ 11.1. Types of errors
                                                                                                              • ☰ 11.2. Reading error messages
                                                                                                                • ☰ 11.3. Common beginner errors
                                                                                                                  • ☰ 11.4. try and except
                                                                                                                    • ☰ 11.5. Debugging strategies
                                                                                                                    • ☰ 12. Chapter 12: Using Python Libraries
                                                                                                                      • ☰ 12.1. What is a library?
                                                                                                                        • ☰ 12.2. Importing modules
                                                                                                                          • ☰ 12.3. Standard libraries
                                                                                                                            • ☰ 12.3.1. math
                                                                                                                              • ☰ 12.3.2. random
                                                                                                                                • ☰ 12.3.3. datetime
                                                                                                                                • ☰ 12.4. Installing external libraries with pip
                                                                                                                                • ☰ 13. Chapter 13: Object-Oriented Programming
                                                                                                                                  • ☰ 13.1. What is object-oriented programming?
                                                                                                                                    • ☰ 13.2. Classes and objects
                                                                                                                                      • ☰ 13.3. The `__init__` method
                                                                                                                                        • ☰ 13.4. Attributes and methods
                                                                                                                                          • ☰ 13.5. Encapsulation
                                                                                                                                            • ☰ 13.6. Inheritance (basics)
                                                                                                                                              • ☰ 13.7. Simple OOP examples
                                                                                                                                              • ☰ 14. Chapter 14: Automation with Python
                                                                                                                                                • ☰ 14.1. What is automation?
                                                                                                                                                  • ☰ 14.2. Automating simple tasks
                                                                                                                                                    • ☰ 14.3. Working with files and folders
                                                                                                                                                      • ☰ 14.4. Automating text processing
                                                                                                                                                        • ☰ 14.5. Scheduling scripts
                                                                                                                                                          • ☰ 14.6. Practical automation examples
                                                                                                                                                          • ☰ 15. Chapter 15: Web Development with Python
                                                                                                                                                            • ☰ 15.1. How the web works (basic overview)
                                                                                                                                                              • ☰ 15.2. Introduction to Flask
                                                                                                                                                                • ☰ 15.3. Creating a simple web server
                                                                                                                                                                  • ☰ 15.4. Routing and templates
                                                                                                                                                                    • ☰ 15.5. Handling user input (forms)
                                                                                                                                                                      • ☰ 15.6. Very basic APIs
                                                                                                                                                                      • ☰ 16. Chapter 16: Data Science with Python
                                                                                                                                                                        • ☰ 16.1. What is data science?
                                                                                                                                                                          • ☰ 16.2. Working with data
                                                                                                                                                                            • ☰ 16.3. Introduction to NumPy
                                                                                                                                                                              • ☰ 16.4. Introduction to pandas
                                                                                                                                                                                • ☰ 16.5. Data visualization with matplotlib
                                                                                                                                                                                  • ☰ 16.6. Simple data analysis examples
                                                                                                                                                                                  • ☰ 17. Chapter 17: Mini Projects
                                                                                                                                                                                    • ☰ 17.1. Number guessing game
                                                                                                                                                                                      • ☰ 17.2. Simple calculator
                                                                                                                                                                                        • ☰ 17.3. To-do list
                                                                                                                                                                                          • ☰ 17.4. Password generator
                                                                                                                                                                                            • ☰ 17.5. Quiz application
                                                                                                                                                                                              • ☰ 17.6. Small web or data project
                                                                                                                                                                                              • ☰ 18. Chapter 18: Next Steps
                                                                                                                                                                                                • ☰ 18.1. How to practice Python
                                                                                                                                                                                                  • ☰ 18.2. Writing clean and readable code
                                                                                                                                                                                                    • ☰ 18.3. Choosing a specialization
                                                                                                                                                                                                      • ☰ 18.3.1. Web development
                                                                                                                                                                                                        • ☰ 18.3.2. Data science
                                                                                                                                                                                                          • ☰ 18.3.3. Automation
                                                                                                                                                                                                            • ☰ 18.3.4. Software development
                                                                                                                                                                                                            • ☰ 18.4. Learning resources and communities
                                                                                                                                                                                                            • ☰ 19. Appendix
                                                                                                                                                                                                              • ☰ 19.1. Python syntax cheat sheet
                                                                                                                                                                                                                • ☰ 19.2. Common errors and solutions
                                                                                                                                                                                                                  • ☰ 19.3. Glossary of terms