Pandas DataFrame: Working with a Whole Table

What is a DataFrame?

A DataFrame is a two-dimensional labelled data structure. Two-dimensional means it has rows and columns, like a table. Each row has a label (the row index) and each column has a name (the column label). Each column is a Series, so different columns can hold different types (text, numbers).

Useful attributes: df.shape (rows, columns), df.index, df.columns, df.dtypes, df.size (number of cells), df.values, df.T (swap rows and columns), df.empty.

Create a DataFrame from a dict of Series, a list of dicts or a CSV

From a dictionary of Series

name  = pd.Series(['Asha', 'Ravi', 'Mohan'])
marks = pd.Series([92, 75, 58])
df = pd.DataFrame({'Name': name, 'Marks': marks})

Dictionary keys become column names. If the Series have different indexes, pandas joins all labels and fills gaps with NaN.

From a list of dictionaries

rows = [{'Name': 'Asha', 'Marks': 92},
        {'Name': 'Ravi', 'Marks': 75, 'City': 'Pune'}]
df = pd.DataFrame(rows)     # each dict is one row; missing City → NaN

From a CSV file

df = pd.read_csv('marks.csv')            # first line = column names
df = pd.read_csv('marks.csv', sep=',', header=0)

You can also make a DataFrame from a dictionary of lists or from a 2D list with columns= and index=.

Displaying a DataFrame

print(df)          # whole table
df.head(3)         # first 3 rows (default 5)
df.tail(2)         # last 2 rows
df.shape           # (5, 3)
df.describe()      # count, mean, min, max … of number columns

Iterating over rows and columns

Iteration means going through items one by one in a loop.

for i, row in df.iterrows():      # row by row
    print(i, row['Name'], row['Marks'])

for col, values in df.items():    # column by column
    print(col, values.tolist())

iterrows() gives (row label, row as a Series). items() (older name iteritems()) gives (column name, column as a Series).

Add, select, delete and rename rows and columns

Columns

df['Grade'] = ['A1', 'B1', 'C2', 'A2', 'C1']   # add (or change) a column
df['Marks']                    # select one column (a Series)
df[['Name', 'Marks']]          # select many columns (a DataFrame)
df = df.drop('Grade', axis=1)  # delete a column
del df['City']                 # another way to delete

Rows

df.loc[5] = ['Neha', 88, 'Pune']   # add a row with label 5
df.loc[2]                          # select a row
df = df.drop(2)                    # delete row labelled 2 (axis=0 is default)

Rename

df = df.rename(columns={'Marks': 'Score'})
df = df.rename(index={0: 'r0'})

drop() and rename() return a new DataFrame; assign it back, or use inplace=True.

Label indexing and boolean indexing

Label-based: loc

df.loc[2, 'City']            # one cell
df.loc[1:3, 'Name':'Marks']  # labels, end included

Position-based: iloc

df.iloc[0:2]        # rows at positions 0, 1 (end not included)
df.iloc[0, 1]       # row 0, column 1

Boolean indexing

df[df.Marks > 80]
df[(df.City == 'Delhi') & (df.Marks > 90)]   # & = and, | = or, use brackets
df.loc[df.Marks < 60, 'Name']

Importing and exporting CSV files

A CSV (comma-separated values) file is plain text: one row per line, values separated by commas. Any spreadsheet can open it.

df = pd.read_csv('marks.csv')                 # import
df.to_csv('result.csv')                       # export with index
df.to_csv('result.csv', index=False)          # export without the index column
df.to_csv('result.txt', sep='\t')            # tab-separated

Try it: your class result sheet

Type 6 rows (Name, Marks, City) in any spreadsheet and save as marks.csv. In Python: read it, print df.shape, add a column Pass = df.Marks >= 33, filter students above 80, and save them with to_csv('toppers.csv', index=False). Open toppers.csv in the spreadsheet. Before each line, predict the answer; then check. Compare with steps 4 and 5 of the 3D.

Key formulas and definitions

• pd.DataFrame(dict_of_series) · pd.DataFrame(list_of_dicts) · pd.read_csv(file)
• df.shape = (rows, columns) · df.size = rows × columns
• loc → labels (end included) · iloc → positions (end not included)
• drop(label, axis=0) row · drop(name, axis=1) column
• df[condition] keeps rows where condition is True · & and, | or
• df.to_csv(file, index=False)

Worked examples

Common mistakes

• Using and / or inside a DataFrame condition; use & and | with brackets around each part.
• Forgetting axis=1 when dropping a column, so pandas looks for a row with that label and gives an error.
• Expecting drop() or rename() to change df without assigning back or using inplace=True.
• Mixing up loc (labels, end included) and iloc (positions, end left out).