相較于《利用Python進行數(shù)據(jù)分析》该互，本書最大的特點是所有操作都變成了分解動作，而且每步都有詳細講解韭畸。但是宇智，書寫的有點啰嗦，而Jupyter Notebook又有些錯胰丁。我對兩者做了整合和總結随橘。
第一遍整理完，還有許多地方不足锦庸，還要再弄。
第01章 Pandas基礎
第02章 DataFrame運算
第03章 數(shù)據(jù)分析入門
第04章 選取數(shù)據(jù)子集
第05章 布爾索引
第06章 索引對齊
第07章 分組聚合、過濾谷婆、轉換
第08章 數(shù)據(jù)清理
第09章 合并Pandas對象
第10章 時間序列分析
第11章 用Matplotlib财松、Pandas、Seaborn進行可視化
作者Theodore Petrou扬卷，Dunder Data創(chuàng)始人
公司網(wǎng)址牙言，http://www.dunderdata.com（dunder是蒸餾朗姆酒的殘留液體，取這個名字是類比數(shù)據(jù)分析過程）
GitHub地址：https://github.com/tdpetrou
領英個人頁面：https://www.linkedin.com/in/tedpetrou
推特：https://twitter.com/tedpetrou?lang=en
Medium博客：https://medium.com/@petrou.theodore
下載代碼：https://github.com/PacktPublishing/Pandas-Cookbook
下載本書 pdf：鏈接
下載本書 mobi：鏈接
# 引入pandas和numpy的約定
in[1]: import pandas as pd
import numpy as np
1. DataFrame的結構
# 設定最大列數(shù)和最大行數(shù)
in[2]: pd.set_option('max_columns', 8, 'max_rows', 10)
# 用read_csv()方法讀取csv文件
# head()方法可以查看前五行怪得，head(n)可以查看前n行
in[3]: movie = pd.read_csv('data/movie.csv')
movie.head()
out[3]:
2. 訪問DataFrame的組件
in[4]: # 提取列索引
columns = movie.columns
# 提取行索引
index = movie.index
# 提取數(shù)據(jù)
data = movie.values
in[5]: columns
out[5]: Index(['color', 'director_name', 'num_critic_for_reviews', 'duration',
'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name',
'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name',
'movie_title', 'num_voted_users', 'cast_total_facebook_likes',
'actor_3_name', 'facenumber_in_poster', 'plot_keywords',
'movie_imdb_link', 'num_user_for_reviews', 'language', 'country',
'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes',
'imdb_score', 'aspect_ratio', 'movie_facebook_likes'],
dtype='object')
in[6]: index
out[6]: RangeIndex(start=0, stop=4916, step=1)
in[7]: data
out[7]: array([['Color', 'James Cameron', 723.0, ..., 7.9, 1.78, 33000],
['Color', 'Gore Verbinski', 302.0, ..., 7.1, 2.35, 0],
['Color', 'Sam Mendes', 602.0, ..., 6.8, 2.35, 85000],
...,
['Color', 'Benjamin Roberds', 13.0, ..., 6.3, nan, 16],
['Color', 'Daniel Hsia', 14.0, ..., 6.3, 2.35, 660],
['Color', 'Jon Gunn', 43.0, ..., 6.6, 1.85, 456]], dtype=object)
in[8]: # index的類型
type(index) # pandas.core.indexes.range.RangeIndex
out[8]: pandas.core.indexes.range.RangeIndex
in[9]: # columns的類型
type(columns) # pandas.core.indexes.base.Index
out[9]: pandas.core.indexes.base.Index
in[10]: # data的類型
type(data) # numpy.ndarray
out[10]: numpy.ndarray
in[11]: # 判斷是不是子類型
issubclass(pd.RangeIndex, pd.Index) # True
out[11]: True
更多
in[12]: # 訪問index的值
index.values
# index的值是個列表咱枉，所以可以索引或切片
index.values[0]
out[12]: array([ 0, 1, 2, ..., 4913, 4914, 4915])
in[13]: # 訪問columns的值
columns.values
out[13]: array(['color', 'director_name', 'num_critic_for_reviews', 'duration',
'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name',
'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name',
'movie_title', 'num_voted_users', 'cast_total_facebook_likes',
'actor_3_name', 'facenumber_in_poster', 'plot_keywords',
'movie_imdb_link', 'num_user_for_reviews', 'language', 'country',
'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes',
'imdb_score', 'aspect_ratio', 'movie_facebook_likes'], dtype=object)
3. 理解數(shù)據(jù)類型
in[14]: movie = pd.read_csv('data/movie.csv')
# 各列的類型
in[15]: movie.dtypes
out[15]: color object
director_name object
num_critic_for_reviews float64
duration float64
director_facebook_likes float64
...
title_year float64
actor_2_facebook_likes float64
imdb_score float64
aspect_ratio float64
movie_facebook_likes int64
Length: 28, dtype: object
in[16]: # 顯示各類型的數(shù)量
movie.get_dtype_counts()
out[16]: float64 13
int64 3
object 12
dtype: int64
4. 選擇一列數(shù)據(jù)卑硫，作為Series
in[17]: movie = pd.read_csv('data/movie.csv')
in[18]: # 選擇director_name這列
movie['director_name']
out[18]: 0 James Cameron
1 Gore Verbinski
2 Sam Mendes
3 Christopher Nolan
4 Doug Walker
...
4911 Scott Smith
4912 NaN
4913 Benjamin Roberds
4914 Daniel Hsia
4915 Jon Gunn
Name: director_name, Length: 4916, dtype: object
in[19]: # 也可以通過屬性的方式選取
movie.director_name
out[19]: 0 James Cameron
1 Gore Verbinski
2 Sam Mendes
3 Christopher Nolan
4 Doug Walker
...
4911 Scott Smith
4912 NaN
4913 Benjamin Roberds
4914 Daniel Hsia
4915 Jon Gunn
Name: director_name, Length: 4916, dtype: object
# 查看類型
in[20]: type(movie['director_name'])
out[20]: pandas.core.series.Series
更多
in[21]: director = movie['director_name']
# 查看選取的列的名字
director.name
out[21]: 'director_name'
in[22]: # 單列Series轉換為DataFrame
director.to_frame().head()
out[22]:
director_name
0 James Cameron
1 Gore Verbinski
2 Sam Mendes
3 Christopher Nolan
4 Doug Walker
5. 調(diào)用Series方法
準備
in[23]: # 查看Series所有不重復的指令
s_attr_methods = set(dir(pd.Series))
# 該集合的大小
len(s_attr_methods)
out[23]: 442
in[24]: # 查看DataFrame所有不重復的指令
df_attr_methods = set(dir(pd.DataFrame))
len(df_attr_methods)
out[24]: 445
in[25]: # 這兩個集合中有多少共有的指令
len(s_attr_methods & df_attr_methods)
out[25]: 376
原理
in[26]: # 選取director和actor_1_fb_likes兩列
movie = pd.read_csv('data/movie.csv')
director = movie['director_name']
actor_1_fb_likes = movie['actor_1_facebook_likes']
# 查看頭部
in[27]: director.head()
out[27]: 0 James Cameron
1 Gore Verbinski
2 Sam Mendes
3 Christopher Nolan
4 Doug Walker
Name: director_name, dtype: object
in[28]: actor_1_fb_likes.head()
out[28]: 0 1000.0
1 40000.0
2 11000.0
3 27000.0
4 131.0
Name: actor_1_facebook_likes, dtype: float64
in[29]: # 分別計數(shù)
pd.set_option('max_rows', 8)
director.value_counts()
out[29]: Steven Spielberg 26
Woody Allen 22
Clint Eastwood 20
Martin Scorsese 20
..
James Nunn 1
Gerard Johnstone 1
Ethan Maniquis 1
Antony Hoffman 1
Name: director_name, Length: 2397, dtype: int64
in[30]: actor_1_fb_likes.value_counts()
out[30]: 1000.0 436
11000.0 206
2000.0 189
3000.0 150
...
216.0 1
859.0 1
225.0 1
334.0 1
Name: actor_1_facebook_likes, Length: 877, dtype: int64
in[31]: director.size
out[31]: 4916
in[32]: director.shape
out[33]: (4916,)
in[33]: len(director)
out[33]: 4916
in[34]: # director有多少非空值
director.count()
out[34]: 4814 # 說明有102個缺失值
in[35]: # actor_1_fb_likes有多少非空值
actor_1_fb_likes.count()
out[35]: 4909
in[36]: # actor_1_fb_likes的中位分位數(shù)
actor_1_fb_likes.quantile()
out[36]: 982.0
in[37]: # 求最小值、最大值庞钢、平均值拔恰、中位數(shù)、標準差基括、總和
actor_1_fb_likes.min(), actor_1_fb_likes.max(), \
actor_1_fb_likes.mean(), actor_1_fb_likes.median(), \
actor_1_fb_likes.std(), actor_1_fb_likes.sum()
out[37]: (0.0, 640000.0, 6494.488490527602, 982.0, 15106.986883848309, 31881444.0)
in[38]: # 打印描述信息
actor_1_fb_likes.describe()
out[38]: count 4909.000000
mean 6494.488491
std 15106.986884
min 0.000000
25% 607.000000
50% 982.000000
75% 11000.000000
max 640000.000000
Name: actor_1_facebook_likes, dtype: float64
in[39]: director.describe()
out[39]: count 4814
unique 2397
top Steven Spielberg
freq 26
Name: director_name, dtype: object
in[40]: actor_1_fb_likes.quantile(.2)
out[41]: 510.0
in[41]: # 各個十分之一分位數(shù)
actor_1_fb_likes.quantile([.1, .2, .3, .4, .5, .6, .7, .8, .9])
out[41]: 0.1 240.0
0.2 510.0
0.3 694.0
0.4 854.0
...
0.6 1000.0
0.7 8000.0
0.8 13000.0
0.9 18000.0
Name: actor_1_facebook_likes, Length: 9, dtype: float64
# 非空值
In[42]: director.isnull()
Out[42]: 0 False
1 False
2 False
3 False
...
4912 True
4913 False
4914 False
4915 False
Name: director_name, Length: 4916, dtype: bool
# 填充缺失值
In[43]: actor_1_fb_likes_filled = actor_1_fb_likes.fillna(0)
actor_1_fb_likes_filled.count()
Out[43]: 4916
# 刪除缺失值
In[44]: actor_1_fb_likes_dropped = actor_1_fb_likes.dropna()
actor_1_fb_likes_dropped.size
Out[44]: 4909
更多
# value_counts(normalize=True) 可以返回頻率
In[45]: director.value_counts(normalize=True)
Out[45]: Steven Spielberg 0.005401
Woody Allen 0.004570
Clint Eastwood 0.004155
Martin Scorsese 0.004155
...
James Nunn 0.000208
Gerard Johnstone 0.000208
Ethan Maniquis 0.000208
Antony Hoffman 0.000208
Name: director_name, Length: 2397, dtype: float64
# 判斷是否有缺失值
In[46]: director.hasnans
Out[46]: True
# 判斷是否是非缺失值
In[47]: director.notnull()
Out[47]: 0 True
1 True
2 True
3 True
...
4912 False
4913 True
4914 True
4915 True
Name: director_name, Length: 4916, dtype: bool
6. 在Series上使用運算符
In[48]: pd.options.display.max_rows = 6
In[49]: 5 + 9 # 加法
Out[49]: 14
In[50]: 4 ** 2 # 冪運算
Out[50]: 16
In[51]: a = 10 # 賦值
In[52]: 5 <= 9 # 小于等于
Out[52]: True
In[53]: 'abcde' + 'fg' # 字符串拼接
Out[53]: 'abcdefg'
In[54]: not (5 <= 9) # 非運算符
Out[54]: False
In[55]: 7 in [1, 2, 6] # in運算符
Out[55]: False
In[56]: set([1,2,3]) & set([2,3,4]) # 求交集
Out[56]: {2, 3}
# 不支持列表和整數(shù)間的運算
In[57]: [1, 2, 3] - 3
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-57-7ca967348b32> in <module>()
----> 1 [1, 2, 3] - 3
TypeError: unsupported operand type(s) for -: 'list' and 'int'
In[58]: a = set([1,2,3])
a[0] # 集合不支持索引
準備
# 選取imdb_score這列
In[59]: movie = pd.read_csv('data/movie.csv')
imdb_score = movie['imdb_score']
imdb_score
Out[59]: 0 7.9
1 7.1
2 6.8
...
4913 6.3
4914 6.3
4915 6.6
Name: imdb_score, Length: 4916, dtype: float64
# 每列值加1
In[60]: imdb_score + 1
Out[60]: 0 8.9
1 8.1
2 7.8
...
4913 7.3
4914 7.3
4915 7.6
Name: imdb_score, Length: 4916, dtype: float64
# 每列值乘以2.5
In[61]: imdb_score * 2.5
Out[61]: 0 19.75
1 17.75
2 17.00
...
4913 15.75
4914 15.75
4915 16.50
Name: imdb_score, Length: 4916, dtype: float64
# 每列值除以7的余數(shù)
In[62]: imdb_score // 7
Out[62]: 0 1.0
1 1.0
2 0.0
...
4913 0.0
4914 0.0
4915 0.0
Name: imdb_score, Length: 4916, dtype: float64
# 判斷是否大于7
In[63]: imdb_score > 7
Out[63]: 0 True
1 True
2 False
...
4913 False
4914 False
4915 False
Name: imdb_score, Length: 4916, dtype: bool
# 判斷是否等于字符串
In[64]: director = movie['director_name']
In[65]: director == 'James Cameron'
Out[65]: 0 True
1 False
2 False
...
4913 False
4914 False
4915 False
Name: director_name, Length: 4916, dtype: bool
更多
# 利用通用函數(shù)實現(xiàn)加法
In[66]: imdb_score.add(1) # imdb_score + 1
Out[66]: 0 8.9
1 8.1
2 7.8
...
4913 7.3
4914 7.3
4915 7.6
Name: imdb_score, Length: 4916, dtype: float64
# 利用通用函數(shù)實現(xiàn)乘法
In[67]: imdb_score.mul(2.5) # imdb_score * 2.5
Out[67]: 0 19.75
1 17.75
2 17.00
...
4913 15.75
4914 15.75
4915 16.50
Name: imdb_score, Length: 4916, dtype: float64
# 利用通用函數(shù)實現(xiàn)底除
In[68]: imdb_score.floordiv(7) # imdb_score // 7
Out[68]: 0 1.0
1 1.0
2 0.0
...
4913 0.0
4914 0.0
4915 0.0
Name: imdb_score, Length: 4916, dtype: float64
# 利用通用函數(shù)實現(xiàn)大于
In[69]: imdb_score.gt(7) # imdb_score > 7
Out[69]: 0 True
1 True
2 False
...
4913 False
4914 False
4915 False
Name: imdb_score, Length: 4916, dtype: bool
# 利用通用函數(shù)實現(xiàn)等于
In[70]: director.eq('James Cameron') # director == 'James Cameron'
Out[70]: 0 True
1 False
2 False
...
4913 False
4914 False
4915 False
Name: director_name, Length: 4916, dtype: bool
# 利用通用函數(shù)實現(xiàn)取模
In[71]: imdb_score.astype(int).mod(5)
Out[71]: 0 2
1 2
2 1
..
4913 1
4914 1
4915 1
Name: imdb_score, Length: 4916, dtype: int64
# a是int對象
In[72]: a = type(1)
In[73]: type(a)
Out[73]: type
# a是pandas.core.series.Series對象
In[74]: a = type(imdb_score)
In[75]: a([1,2,3])
Out[75]: 0 1
1 2
2 3
dtype: int64
7. 串聯(lián)Series方法
# value_counts().head(3)颜懊，計數(shù)，查看前三
In[76]: movie = pd.read_csv('data/movie.csv')
actor_1_fb_likes = movie['actor_1_facebook_likes']
director = movie['director_name']
In[77]: director.value_counts().head(3)
Out[77]: Steven Spielberg 26
Woody Allen 22
Clint Eastwood 20
Name: director_name, dtype: int64
# 統(tǒng)計缺失值的數(shù)量
In[78]: actor_1_fb_likes.isnull().sum()
Out[78]: 7
# actor_1_fb_likes的數(shù)據(jù)類型
In[79]: actor_1_fb_likes.dtype
Out[79]: dtype('float64')
# 缺失值填充為0风皿、轉換為整型河爹、查看前五
In[80]: actor_1_fb_likes.fillna(0)\
.astype(int)\
.head()
Out[80]: 0 1000
1 40000
2 11000
3 27000
4 131
Name: actor_1_facebook_likes, dtype: int64
更多
# 缺失值的比例
In[81]: actor_1_fb_likes.isnull().mean()
Out[81]: 0.0014239218877135883
# 使用括號串聯(lián)
In[82]: (actor_1_fb_likes.fillna(0)
.astype(int)
.head())
Out[82]: 0 1000
1 40000
2 11000
3 27000
4 131
Name: actor_1_facebook_likes, dtype: int64
8. 使索引有意義
# set_index()給行索引命名
In[83]: movie = pd.read_csv('data/movie.csv')
In[84]: movie.shape
Out[84]: (4916, 28)
In[85]: movie2 = movie.set_index('movie_title')
movie2
Out[85]:
# 通過index_col參數(shù)命名
In[86]: pd.read_csv('data/movie.csv', index_col='movie_title')
Out[86]:
更多
# 復原索引
In[87]: movie2.reset_index()
9. 重命名行名和列名
# 通過rename()重命名
In[88]: movie = pd.read_csv('data/movie.csv', index_col='movie_title')
In[89]: idx_rename = {'Avatar':'Ratava', 'Spectre': 'Ertceps'}
col_rename = {'director_name':'Director Name',
'num_critic_for_reviews': 'Critical Reviews'}
In[90]: movie.rename(index=idx_rename,
columns=col_rename).head()
Out[90]:
更多
In[91]: movie = pd.read_csv('data/movie.csv', index_col='movie_title')
index = movie.index
columns = movie.columns
index_list = index.tolist()
column_list = columns.tolist()
index_list[0] = 'Ratava'
index_list[2] = 'Ertceps'
column_list[1] = 'Director Name'
column_list[2] = 'Critical Reviews'
In[92]: print(index_list[:5])
['Ratava', "Pirates of the Caribbean: At World's End", 'Ertceps', 'The Dark Knight Rises', 'Star Wars: Episode VII - The Force Awakens']
In[93]: print(column_list)
['color', 'Director Name', 'Critical Reviews', 'duration', 'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name', 'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name', 'num_voted_users', 'cast_total_facebook_likes', 'actor_3_name', 'facenumber_in_poster', 'plot_keywords', 'movie_imdb_link', 'num_user_for_reviews', 'language', 'country', 'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes', 'imdb_score', 'aspect_ratio', 'movie_facebook_likes']
In[94]: movie.index = index_list
movie.columns = column_list
In[95]: movie.head()
Out[95]:
10. 創(chuàng)建、刪除列
# 通過[列名]添加新列
In[96]: movie = pd.read_csv('data/movie.csv')
In[97]: movie['has_seen'] = 0
In[98]: movie.columns
Out[98]: Index(['color', 'director_name', 'num_critic_for_reviews', 'duration',
'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name',
'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name',
'movie_title', 'num_voted_users', 'cast_total_facebook_likes',
'actor_3_name', 'facenumber_in_poster', 'plot_keywords',
'movie_imdb_link', 'num_user_for_reviews', 'language', 'country',
'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes',
'imdb_score', 'aspect_ratio', 'movie_facebook_likes', 'has_seen'],
dtype='object')
# 給新列賦值
In[99]: movie['actor_director_facebook_likes'] = (movie['actor_1_facebook_likes'] +
movie['actor_2_facebook_likes'] +
movie['actor_3_facebook_likes'] +
movie['director_facebook_likes'])
In[100]: movie['actor_director_facebook_likes'].isnull().sum()
Out[100]: 122
# 用all()檢查是否所有的布爾值都為True
In[101]: movie['actor_director_facebook_likes'] = movie['actor_director_facebook_likes'].fillna(0)
In[102]: movie['is_cast_likes_more'] = (movie['cast_total_facebook_likes'] >=
movie['actor_director_facebook_likes'])
In[103]: movie['is_cast_likes_more'].all()
Out[103]: False
In[104]: movie = movie.drop('actor_director_facebook_likes', axis='columns')
In[105]: movie['actor_total_facebook_likes'] = (movie['actor_1_facebook_likes'] +
movie['actor_2_facebook_likes'] +
movie['actor_3_facebook_likes'])
movie['actor_total_facebook_likes'] = movie['actor_total_facebook_likes'].fillna(0)
In[106]: movie['is_cast_likes_more'] = movie['cast_total_facebook_likes'] >= \
movie['actor_total_facebook_likes']
movie['is_cast_likes_more'].all()
Out[106]: True
In[107]: movie['pct_actor_cast_like'] = (movie['actor_total_facebook_likes'] /
movie['cast_total_facebook_likes'])
In[108]: movie['pct_actor_cast_like'].min(), movie['pct_actor_cast_like'].max()
Out[108]: (0.0, 1.0)
In[109]: movie.set_index('movie_title')['pct_actor_cast_like'].head()
Out[109]: movie_title
Avatar 0.577369
Pirates of the Caribbean: At World's End 0.951396
Spectre 0.987521
The Dark Knight Rises 0.683783
Star Wars: Episode VII - The Force Awakens 0.000000
Name: pct_actor_cast_like, dtype: float64
更多
# 用insert()方法原地插入列
In[110]: profit_index = movie.columns.get_loc('gross') + 1
profit_index
In[111]: movie.insert(loc=profit_index,
column='profit',
value=movie['gross'] - movie['budget'])
In[112]: movie.head()
Out[112]:
第01章 Pandas基礎
第02章 DataFrame運算
第03章 數(shù)據(jù)分析入門
第04章 選取數(shù)據(jù)子集
第05章 布爾索引
第06章 索引對齊
第07章 分組聚合桐款、過濾咸这、轉換
第08章 數(shù)據(jù)清理
第09章 合并Pandas對象
第10章 時間序列分析
第11章 用Matplotlib、Pandas魔眨、Seaborn進行可視化
