Working with data using Pandas

Posted on by Panther

Python has been a popular language for data analysis and manipulation over the years due to its powerful libraries. One of these libraries is Pandas, which is widely used for data analysis. Pandas provides an easy-to-use data structure and data manipulation tools. In this tutorial, we will cover the basics of working with data using Pandas.

Setting Up Pandas

Before we can start working with Pandas, we need to install it. You can install Pandas using pip, the package installer for Python:

pip install pandas

Once installed, you can import it using the following command:

import pandas as pd

The Pandas Data Structure

Pandas provides two fundamental data structures:

  • Series – a one-dimensional array-like object that can hold any data type.
  • DataFrame – a two-dimensional table consisting of rows and columns.

Series Data Structure

A Series can be created by passing a list of values, an array, or a scalar value. The first column represents the index, and the second column represents the values.

import pandas as pd
import numpy as np

data = pd.Series([0.25, 0.5, 0.75, 1.0])
print(data)

Output:

0    0.25
1    0.50
2    0.75
3    1.00
dtype: float64

DataFrame Data Structure

A DataFrame can be created by passing a dictionary of arrays, lists, or Series. The dictionary keys represent the column names, and the dictionary values represent the column data.

data = {'name': ['John', 'Jane', 'Alice', 'Bob'],
        'age': [30, 25, 40, 35],
        'gender': ['male', 'female', 'female', 'male']}

df = pd.DataFrame(data)
print(df)

Output:

    name  age  gender
0   John   30    male
1   Jane   25  female
2  Alice   40  female
3    Bob   35    male

Reading and Writing Data

Pandas provides many functions to read and write data in different formats such as CSV, Excel, SQL, and others.

Reading Data

Pandas provides a wide range of functions to read data:

  • pd.read_csv() – reads a CSV file.
  • pd.read_excel() – reads an Excel file.
  • pd.read_sql() – reads data from a SQL database.

For instance, to read a CSV file, you can use pd.read_csv() as follows:

data = pd.read_csv('data.csv')

Writing Data

Similarly, Pandas provides functions to write data in various formats:

  • df.to_csv() – write a DataFrame to a CSV file.
  • df.to_excel() – write a DataFrame to an Excel file.
  • df.to_sql() – writes data to a SQL database.

For example, to write a DataFrame to a CSV file, you can use df.to_csv() as follows:

df.to_csv('output.csv', index=False)

The index=False parameter will exclude the index column from the CSV file.

Basic Operations

Once we have loaded data into our DataFrame, we can perform various operations on it. Here, we will look at some of the basic operations that we can perform.

Viewing Data

Pandas provides several ways to view data:

  • df.head() – displays the first few rows of the DataFrame.
  • df.tail() – displays the last few rows of the DataFrame.
  • df.index – displays the index of the DataFrame.
  • df.columns – displays the column names of the DataFrame.
  • df.shape – displays the number of rows and columns of the DataFrame.
print(df.head())

Output:

    name  age  gender
0   John   30    male
1   Jane   25  female
2  Alice   40  female
3    Bob   35    male

Selection and Slicing

We can select, filter, and slice data using several methods:

  • df['column_name'] or df.column_name – select a column from the DataFrame.
  • df.loc[row_label, col_label] – select a subset of rows and columns using the row and column labels.
  • df.iloc[row_num, col_num] – select a subset of rows and columns using integer indexing.
  • df.query() – select rows based on a condition.
  • df.filter() – select columns based on a condition.
print(df['name'])

Output:

0     John
1     Jane
2    Alice
3      Bob
Name: name, dtype: object

print(df.loc[0:1, ['name', 'gender']])

Output:

   name  gender
0  John    male
1  Jane  female

Filtering

We can also filter data for specific values or conditions:

print(df[df.age > 30])

Output:

    name  age gender
2  Alice   40      f
3    Bob   35      m

Grouping

We can group our data based on one or more variables and then perform aggregation functions, such as mean, sum, and count, on the grouped data:

grouped_data = df.groupby(['gender'])['age'].mean()
print(grouped_data)

Output:

gender
female    32.5
male      32.5
Name: age, dtype: float64

Conclusion

In this tutorial, we have covered the basics of working with data using Pandas. We learned about the Pandas data structure, reading and writing data, and performing basic operations such as selection, filtering, and grouping. With this knowledge, you can analyze and manipulate any dataset using Pandas.

Related Post