This is a short introduction to pandas, geared mainly for new users。
Customarily,we import as follows:
In [1]: import numpy as np
In [2]: import pandas as pd
Object creation
Creating a Series by passing a list of values, letting pandas create a default integer index
s = pd.Series([1, 3, 5, np.nan, 6, 8])
Creating a DataFrame by passing a NumPy array, with a datetime index and labeled columns
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
index
columns
Creating a DataFrame by passing a dict of objects that can be converted to series-like.
Viewing data
Here is how to view the top and bottom rows of the frame:
df.head()
df.tail(3)
df.index
df.columns
df.describe()
df.T
Sorting by an axis
df.sort_index(axis=1, ascending=False)
df.sort_values(by='B')
Selection
While standard Python / Numpy expressions for selecting and setting are intuitive 直觀的 and come in handy for interactive 互動的 work, for production code, we recommend the optimized pandas data access methods, .at, .iat, .loc and .iloc.
Getting
Selecting a single column, which yields a Series, equivalent 等效的 to df.A:df['A']
Selecting via [], which slices the rows. 切片
df[0:3]
df['20130102':'20130104']
Selection by label
df.loc[dates[0]]
df.loc[:, ['A', 'B']]
df.loc['20130102':'20130104', ['A', 'B']]
df.loc['20130102', ['A', 'B']]
df.loc[dates[0], 'A']
df.at[dates[0], 'A']
Selection by position
df.iloc[3]
df.iloc[3:5, 0:2]
df.iloc[[1, 2, 4], [0, 2]]
df.iloc[1:3, :]
df.iloc[:, 1:3]
df.iloc[1, 1]
df.iat[1, 1]
Boolean indexing
df[df['A'] > 0]
df[df > 0]
df2[df2['E'].isin(['two', 'four'])]
Setting
Setting a new column automatically aligns the data by the indexes.
Setting values by label:
Setting values by position:
Setting by assigning with a NumPy array:
Missing data
pandas primarily uses the value np.nan to represent missing data. It is by default not included in computations
Reindexing allows you to change/add/delete the index on a specified axis. This returns a copy of the data.
To drop any rows that have missing data.
df1.dropna(how='any')
Filling missing data.
df1.fillna(value=5)
To get the boolean mask where values are nan.
pd.isna(df1)
Operations
Stats
Operations in general exclude missing data.
df.mean()
Apply
df.apply(np.cumsum)
df.apply(lambda x: x.max() - x.min())
Histogramming
s.value_counts()
String Methods
s.str.lower()
Merge
Concatenating pandas objects together with concat()
pd.concat(df1,df2)
pd.merge(left, right, on='key')
Grouping
By“group by” we are referring to a process involving one or more of the followingsteps:
Splitting the data into groups based on some criteria
Applying a function to each group independently
Combining the results into a data structure
df.groupby('A').sum()
df.groupby(['A', 'B']).sum()
Reshaping
The stack() method “compresses” a level in the DataFrame’s columns.
With a “stacked” DataFrame or Series (having a MultiIndex as the index), the inverse operation of stack() is unstack(), which by default unstacks the last level
stacked.unstack()
stacked.unstack(1)
Pivot tables
pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'])
Time series
rng = pd.date_range('1/1/2012', periods=100, freq='S')
ts_utc = ts.tz_localize('UTC')
rng = pd.date_range('1/1/2012', periods=5, freq='M')
ps = ts.to_period()
ps.to_timestamp()
Categoricals
df["grade"]=df["raw_grade"].astype("category")
df.groupby("grade").size()
Plotting
We use the standard convention for referencing the matplotlib API
import matplotlib.pyplot as plt
plt.close('all')
On a DataFrame, the plot() method is a convenience to plot all of the columns with labels
plt.figure()
df.plot()
Getting data in/out
CSV
Writing to a csv file.
df.to_csv('foo.csv')
Reading from a csv file.
pd.read_csv('foo.csv')
HDF5
df.to_hdf('foo.h5', 'df')
pd.read_hdf('foo.h5', 'df')
Excel
df.to_excel('foo.xlsx', sheet_name='Sheet1')
pd.read_excel('foo.xlsx', 'Sheet1', index_col=None, na_values=['NA']