【Chapter 7.2 】數(shù)據(jù)轉(zhuǎn)換
本章到目前為止介紹的都是數(shù)據(jù)的重排键闺。另一類重要操作則是過濾、清理以及其他的轉(zhuǎn)換工作。
1 刪除重復(fù)值
data = pd.DataFrame({'k1': ['one', 'two'] * 3 + ['two'],
'k2': [1, 1, 2, 3, 3, 4, 4]})
data
Out[468]:
k1 k2
0 one 1
1 two 1
2 one 2
3 two 3
4 one 3
5 two 4
6 two 4
DataFrame方法duplicated返回的是一個boolean Series创坞,表示一個row是否是重復(fù)的(根據(jù)前一行來判斷):
data.duplicated()
Out[469]:
0 False
1 False
2 False
3 False
4 False
5 False
6 True
dtype: bool
drop_duplicateds返回一個DataFrame平酿,會刪除重復(fù)的部分:
data.drop_duplicates()
Out[470]:
k1 k2
0 one 1
1 two 1
2 one 2
3 two 3
4 one 3
5 two 4
上面兩種方法都默認(rèn)考慮所有列凤优;另外,我們可以指定一部分來檢測重復(fù)值蜈彼。假設(shè)我們只想檢測'k1'列的重復(fù)值:
data['v1'] = range(7)
data
Out[471]:
k1 k2 v1
0 one 1 0
1 two 1 1
2 one 2 2
3 two 3 3
4 one 3 4
5 two 4 5
6 two 4 6
data.drop_duplicates(['k1'])
Out[472]:
k1 k2 v1
0 one 1 0
1 two 1 1
duplicated和drop_duplicated默認(rèn)保留第一次觀測到的數(shù)值組合筑辨。設(shè)置keep='last'
能返回最后一個:
data.drop_duplicates(['k1'])
Out[472]:
k1 k2 v1
0 one 1 0
1 two 1 1
data.drop_duplicates(['k1', 'k2'], keep='last')
Out[473]:
k1 k2 v1
0 one 1 0
1 two 1 1
2 one 2 2
3 two 3 3
4 one 3 4
6 two 4 6
data.drop_duplicates(['k1'],keep = 'last')
Out[474]:
k1 k2 v1
4 one 3 4
6 two 4 6
2 用函數(shù)和映射來轉(zhuǎn)換數(shù)據(jù)
對于許多數(shù)據(jù)集,你可能希望根據(jù)數(shù)組幸逆、Series或DataFrame列中的值來實現(xiàn)轉(zhuǎn)換工作棍辕。我們來看看下面這組有關(guān)肉類的數(shù)據(jù):
data = pd.DataFrame({'food': ['bacon', 'pulled pork', 'bacon',
'Pastrami', 'corned beef', 'Bacon',
'pastrami', 'honey ham', 'nova lox'],
'ounces': [4, 3, 12, 6, 7.5, 8, 3, 5, 6]})
data
Out[475]:
food ounces
0 bacon 4.0
1 pulled pork 3.0
2 bacon 12.0
3 Pastrami 6.0
4 corned beef 7.5
5 Bacon 8.0
6 pastrami 3.0
7 honey ham 5.0
8 nova lox 6.0
假設(shè)你想加一列暮现,表明每種肉來源的動物是什么。我們可以寫一個映射:
meat_to_animal = {
'bacon': 'pig',
'pulled pork': 'pig',
'pastrami': 'cow',
'corned beef': 'cow',
'honey ham': 'pig',
'nova lox': 'salmon'
}
用于series的map方法接受一個函數(shù)楚昭,或是一個字典栖袋,包含著映射關(guān)系,但這里有一個小問題抚太,有些肉是大寫塘幅,有些是小寫。因此尿贫,我們先用str.lower把所有的值變?yōu)樾?
lowercased = data['food'].str.lower()
lowercased
Out[477]:
0 bacon
1 pulled pork
2 bacon
3 pastrami
4 corned beef
5 bacon
6 pastrami
7 honey ham
8 nova lox
Name: food, dtype: object
data['animal'] = lowercased.map(meat_to_animal)
data
Out[478]:
food ounces animal
0 bacon 4.0 pig
1 pulled pork 3.0 pig
2 bacon 12.0 pig
3 Pastrami 6.0 cow
4 corned beef 7.5 cow
5 Bacon 8.0 pig
6 pastrami 3.0 cow
7 honey ham 5.0 pig
8 nova lox 6.0 salmon
我們也可以用一個函數(shù)解決上面的問題:
data['food'].map(lambda x: meat_to_animal[x.lower()])
Out[479]:
0 pig
1 pig
2 pig
3 cow
4 cow
5 pig
6 cow
7 pig
8 salmon
Name: food, dtype: object
使用map是一個很簡便的方法电媳,用于element-wise轉(zhuǎn)換和其他一些數(shù)據(jù)清洗操作。
3 Replacing Values(替換值)
其實fillna是一個特殊換的替換操作庆亡。map可以用于修改一個object里的部分值匆背,但是replace能提供一個更簡單和更靈活的方法做到這點。下面是一個series:
data = pd.Series([1., -999., 2., -999., -1000., 3.])
data
Out[480]:
0 1.0
1 -999.0
2 2.0
3 -999.0
4 -1000.0
5 3.0
dtype: float64
3 替換值
其實fillna是一個特殊換的替換操作身冀。map可以用于修改一個object里的部分值钝尸,但是replace能提供一個更簡單和更靈活的方法做到這點。下面是一個series:
data = pd.Series([1., -999., 2., -999., -1000., 3.])
data
Out[481]:
0 1.0
1 -999.0
2 2.0
3 -999.0
4 -1000.0
5 3.0
dtype: float64
這里-999可能是用來表示缺失值的標(biāo)識符搂根。用NA來替代的話珍促,用replace,會產(chǎn)生一個新series(除非使用inplace=True):
data.replace(-999, np.nan)
Out[482]:
0 1.0
1 NaN
2 2.0
3 NaN
4 -1000.0
5 3.0
dtype: float64
-999這個值可能是一個表示缺失數(shù)據(jù)的標(biāo)記值剩愧。要將其替換為pandas能夠理解的NA值猪叙,我們可以利用replace來產(chǎn)生一個新的Series(除非傳入inplace=True):
data.replace(-999, np.nan)
Out[483]:
0 1.0
1 NaN
2 2.0
3 NaN
4 -1000.0
5 3.0
dtype: float64
如果你希望一次性替換多個值,可以傳入一個由待替換值組成的列表以及一個替換值::
data.replace([-999, -1000], np.nan)
Out[484]:
0 1.0
1 NaN
2 2.0
3 NaN
4 NaN
5 3.0
dtype: float64
對于不同的值用不同的替換值仁卷,也是導(dǎo)入一個list:
data.replace([-999, -1000], [np.nan, 0])
Out[485]:
0 1.0
1 NaN
2 2.0
3 NaN
4 0.0
5 3.0
dtype: float64
參數(shù)也可以是一個dict:
data.replace({-999: np.nan, -1000: 0})
Out[486]:
0 1.0
1 NaN
2 2.0
3 NaN
4 0.0
5 3.0
dtype: float64
注意:data.replace方法和data.str.replace方法是不同的穴翩,后者會對string進(jìn)行element-wise替換。
4 Renaming Axis Indexes(重命名Axis Indexes)
像是series里的value一樣锦积,axis label也能類似地是函數(shù)或映射來轉(zhuǎn)換芒帕,產(chǎn)生一個新的object。當(dāng)然也可以設(shè)置in-place不產(chǎn)生新的數(shù)據(jù):
data = pd.DataFrame(np.arange(12).reshape((3, 4)),
index=['Ohio', 'Colorado', 'New York'],
columns=['one', 'two', 'three', 'four'])
data
Out[487]:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
New York 8 9 10 11
與series相同丰介,axis index有一個map方法:
transform = lambda x: x[:4].upper()
transform
Out[488]: <function __main__.<lambda>>
data.index
Out[489]: Index(['Ohio', 'Colorado', 'New York'], dtype='object')
#可以賦值給index背蟆,以in-place的方式修改DataFrame:
data.index.map(transform)
Out[490]: Index(['OHIO', 'COLO', 'NEW '], dtype='object')
如果想要創(chuàng)建數(shù)據(jù)集的轉(zhuǎn)換版(而不是修改原始數(shù)據(jù)),比較實用的方法是rename:
data.rename(index=str.title, columns=str.upper)
Out[491]:
ONE TWO THREE FOUR
Ohio 0 1 2 3
Colorado 4 5 6 7
New York 8 9 10 11
注意哮幢,rename能用于dict一樣的oject带膀,
data.rename(index={'OHIO': 'INDIANA'},
columns={'three': 'pekaboo'})
Out[492]:
one two pekaboo four
Ohio 0 1 2 3
Colorado 4 5 6 7
New York 8 9 10 11
rename能讓你避免陷入手動賦值給index和columns的雜務(wù)中〕裙福可以用inplace直接修改原始數(shù)據(jù):
data.rename(index={'OHIO': 'INDIANA'}, inplace=True)
data
Out[493]:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
New York 8 9 10 11
5 Discretization and Binning(離散化和裝箱)
連續(xù)型數(shù)據(jù)經(jīng)常被離散化或分散成bins(分箱)來分析垛叨。假設(shè)你有一組數(shù)據(jù),你想把人分到不同的年齡組里:
ages = [20, 22, 25, 27, 21, 23, 37, 31, 61, 45, 41, 32]
我們把這些分到四個bin里柜某,19~25, 26~35, 36~60, >60嗽元×哺伲可以用pandas里的cut:
bins = [18, 25, 35, 60, 100]
cats = pd.cut(ages, bins)
cats
[(18, 25], (18, 25], (18, 25], (25, 35], (18, 25], ..., (25, 35], (60, 100], (35, 60], (35, 60], (25, 35]]
Length: 12
Categories (4, interval[int64]): [(18, 25] < (25, 35] < (35, 60] < (60, 100]]
返回的是一個特殊的Categorical object。我們看到的結(jié)果描述了pandas.cut如何得到bins还棱≡卮龋可以看作是一個string數(shù)組用來表示bin的名字惭等,它內(nèi)部包含了一個categories數(shù)組珍手,用來記錄不同類別的名字,并伴有表示ages的label(可以通過codes屬性查看):
cats.codes #4種情況
Out[496]: array([0, 0, 0, 1, 0, 0, 2, 1, 3, 2, 2, 1], dtype=int8)
cats.categories
Out[497]:
IntervalIndex([(18, 25], (25, 35], (35, 60], (60, 100]]
closed='right',
dtype='interval[int64]')
pd.value_counts(cats)
Out[498]:
(18, 25] 5
(35, 60] 3
(25, 35] 3
(60, 100] 1
這里pd.value_counts(cats)是pandas.cut后bin的數(shù)量辞做。
這里我們注意一下區(qū)間琳要。括號表示不包含,方括號表示包含秤茅。你可以自己設(shè)定哪一邊關(guān)閉(right=False):
pd.cut(ages, [18, 26, 36, 61, 100], right=False)
Out[499]:
[[18, 26), [18, 26), [18, 26), [26, 36), [18, 26), ..., [26, 36), [61, 100), [36, 61), [36, 61), [26, 36)]
Length: 12
Categories (4, interval[int64]): [[18, 26) < [26, 36) < [36, 61) < [61, 100)]
你也可以用一個list或數(shù)組給labels選項來設(shè)定bin的名字:
group_names = ['Youth', 'YoungAdult', 'MiddleAged', 'Senior']
pd.cut(ages, bins, labels=group_names)
Out[501]:
[Youth, Youth, Youth, YoungAdult, Youth, ..., YoungAdult, Senior, MiddleAged, MiddleAged, YoungAdult]
Length: 12
Categories (4, object): [Youth < YoungAdult < MiddleAged < Senior]
如果你只是給一個bins的數(shù)量來cut稚补,而不是自己設(shè)定每個bind的范圍,cut會根據(jù)最大值和最小值來計算等長的bins框喳。比如下面我們想要做一個均勻分布的四個bins:
data = np.random.rand(20)
pd.cut(data, 4, precision=2)
Out[503]:
[(0.022, 0.26], (0.022, 0.26], (0.26, 0.5], (0.26, 0.5], (0.26, 0.5], ..., (0.5, 0.74], (0.26, 0.5], (0.74, 0.98], (0.022, 0.26], (0.74, 0.98]]
Length: 20
Categories (4, interval[float64]): [(0.022, 0.26] < (0.26, 0.5] < (0.5, 0.74] < (0.74, 0.98]]
precision=2選項表示精確到小數(shù)點后兩位课幕。
一個近似的函數(shù),qcut五垮,會按照數(shù)據(jù)的分位數(shù)來分箱乍惊。取決于數(shù)據(jù)的分布,用cut通常不能保證每一個bin有一個相同數(shù)量的數(shù)據(jù)點放仗。而qcut是按百分比來切的润绎,所以可以得到等數(shù)量的bins:
data = np.random.randn(1000) # Normally distributed
cats = pd.qcut(data, 4) # Cut into quartiles
cats
Out[505]:
[(-0.0426, 0.637], (-0.0426, 0.637], (0.637, 3.835], (-3.259, -0.725], (-0.0426, 0.637], ..., (-3.259, -0.725], (0.637, 3.835], (-0.725, -0.0426], (-3.259, -0.725], (-3.259, -0.725]]
Length: 1000
Categories (4, interval[float64]): [(-3.259, -0.725] < (-0.725, -0.0426] < (-0.0426, 0.637] < (0.637, 3.835]]
pd.value_counts(cats)
Out[506]:
(0.637, 3.835] 250
(-0.0426, 0.637] 250
(-0.725, -0.0426] 250
(-3.259, -0.725] 250
類似的,在cut中我們可以自己指定百分比:
cats2 = pd.cut(data, [0, 0.1, 0.5, 0.9, 1.]) # 累進(jìn)的百分比
cats2
Out[507]:
[(0.1, 0.5], (0.1, 0.5], NaN, NaN, (0.1, 0.5], ..., NaN, NaN, NaN, NaN, NaN]
Length: 1000
Categories (4, interval[float64]): [(0.0, 0.1] < (0.1, 0.5] < (0.5, 0.9] < (0.9, 1.0]]
pd.value_counts(cats2)
Out[508]:
(0.1, 0.5] 154
(0.5, 0.9] 127
(0.0, 0.1] 33
(0.9, 1.0] 30
dtype: int64
在之后的章節(jié)我們還會用到cut和qcut诞挨,這些離散函數(shù)對于量化和群聚分析很有用莉撇。
6 Detecting and Filtering Outliers(檢測和過濾異常值)
過濾或轉(zhuǎn)換異常值是數(shù)組操作的一個重頭戲。下面的DataFrame有正態(tài)分布的數(shù)據(jù):
data = pd.DataFrame(np.random.randn(1000, 4))
data.describe()
Out[509]:
0 1 2 3
count 1000.000000 1000.000000 1000.000000 1000.000000
mean -0.025633 -0.062057 0.007312 -0.037653
std 1.029759 0.989084 0.999814 1.031975
min -3.808553 -3.442724 -2.647014 -4.198109
25% -0.735954 -0.713492 -0.691054 -0.742947
50% -0.068758 -0.079244 -0.048442 0.019906
75% 0.619517 0.628789 0.728752 0.661834
max 3.822244 3.775342 3.229284 3.698836
假設(shè)我們想要找一個列中惶傻,絕對值大于3的數(shù)字:
data.head()
Out[510]:
0 1 2 3
0 -0.046264 0.420009 -0.244728 -1.907207
1 1.207292 -0.163423 0.043483 0.543991
2 -0.902596 -1.747752 -0.217981 -2.412121
3 0.233481 -0.741199 0.330361 -0.478363
4 -0.621485 -0.850046 0.795616 1.558565
col = data[2]
col.head()
Out[511]:
0 -0.244728
1 0.043483
2 -0.217981
3 0.330361
4 0.795616
Name: 2, dtype: float64
col[np.abs(col) > 3]
Out[512]:
141 3.115218
153 3.216137
384 3.229284
Name: 2, dtype: float64
選中所有絕對值大于3的行棍郎,可以用any方法在一個boolean DataFrame上:
data[(np.abs(data) > 3)].head()
Out[513]:
0 1 2 3
0 NaN NaN NaN NaN
1 NaN NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN NaN NaN NaN
4 NaN NaN NaN NaN
data[(np.abs(data) > 3).any(1)] # any中axis=1表示column
Out[514]:
0 1 2 3
141 -0.863282 -1.357953 3.115218 -0.298364
153 -0.090615 0.307913 3.216137 -1.518234
201 -1.221955 3.775342 0.719315 1.125198
220 -1.204790 -3.351440 -0.562853 -0.183570
271 -0.498556 -1.282907 0.387269 -4.198109
.. ... ... ... ...
480 3.297639 -1.583162 0.037219 -0.883607
551 3.822244 -1.058748 0.759103 1.042670
558 -3.808553 -1.112588 -1.458445 -1.363454
597 -0.839910 1.555326 0.885032 3.698836
798 -3.382429 -0.283277 -1.388250 0.855298
下面是把絕對值大于3的數(shù)字直接變成-3或3:
data[np.abs(data) > 3] = np.sign(data) * 3
data[21:23]
Out[516]:
0 1 2 3
21 2.043230 -1.900945 0.284129 0.357881
22 0.443324 0.377316 -0.034172 1.592983
data.describe()
Out[517]:
0 1 2 3
count 1000.000000 1000.000000 1000.000000 1000.000000
mean -0.025562 -0.062038 0.006751 -0.037154
std 1.022234 0.983840 0.998077 1.025519
min -3.000000 -3.000000 -2.647014 -3.000000
25% -0.735954 -0.713492 -0.691054 -0.742947
50% -0.068758 -0.079244 -0.048442 0.019906
75% 0.619517 0.628789 0.728752 0.661834
max 3.000000 3.000000 3.000000 3.000000
np.sign(data)會根據(jù)值的正負(fù)號來得到1或-1:
np.sign(data).head()
Out[518]:
0 1 2 3
0 -1.0 1.0 -1.0 -1.0
1 1.0 -1.0 1.0 1.0
2 -1.0 -1.0 -1.0 -1.0
3 1.0 -1.0 1.0 -1.0
4 -1.0 -1.0 1.0 1.0
7 Permutation and Random Sampling(排列和隨機采樣)
排列(隨機排序)一個series或DataFrame中的row,用numpy.random.permutation函數(shù)很容易就能做到银室。調(diào)用permutation的時候設(shè)定好你想要進(jìn)行排列的axis坝撑,會產(chǎn)生一個整數(shù)數(shù)組表示新的順序:
df = pd.DataFrame(np.arange(5 * 4).reshape((5, 4)))
df
0 1 2 3
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
3 12 13 14 15
4 16 17 18 19
sampler = np.random.permutation(5)
sampler
Out[519]: array([4, 2, 3, 0, 1])
這個數(shù)組能被用在基于iloc上的indexing或take函數(shù):
df.take(sampler)
Out[520]:
0 1 2 3
4 16 17 18 19
3 12 13 14 15
2 8 9 10 11
1 4 5 6 7
0 0 1 2 3
為了選中一個隨機的子集,而且沒有代替功能(既不影響原來的值粮揉,返回一個新的series或DataFrame)巡李,可以用sample方法:
df.sample(n=3)
0 1 2 3
0 0 1 2 3
3 12 13 14 15
4 16 17 18 19
如果想要生成的樣本帶有替代功能(即允許重復(fù)),給sample中設(shè)定replace=True:
choices = pd.Series([5, 7, -1, 6, 4])
draws = choices.sample(n=10, replace=True)
draws
4 4
4 4
1 7
1 7
1 7
1 7
4 4
3 6
1 7
1 7
dtype: int64
8 Computing Indicator/Dummy Variables(計算指示器/虛擬變量)
Dummy Variables:虛擬變量扶认,又稱虛設(shè)變量侨拦、名義變量或啞變量,用以反映質(zhì)的屬性的一個人工變量,是量化了的自變量,通常取值為0或1。
另一種在統(tǒng)計模型上的轉(zhuǎn)換或機器學(xué)習(xí)應(yīng)用是把一個categorical variable(類別變量)變?yōu)橐粋€dummy or indicator matrix(虛擬或指示器矩陣)辐宾。如果DataFrame中的一列有k個不同的值狱从,我們可以用一個矩陣或DataFrame用k列來表示膨蛮,1或0。pandas有一個get_dummies函數(shù)實現(xiàn)這個工作季研,當(dāng)然敞葛,你自己設(shè)計一個其實也不難。這里舉個例子:
df = pd.DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'b'],
'data1': range(6)})
df
pd.get_dummies(df['key'])
a b c
0 0.0 1.0 0.0
1 0.0 1.0 0.0
2 1.0 0.0 0.0
3 0.0 0.0 1.0
4 1.0 0.0 0.0
5 0.0 1.0 0.0
在一些情況里与涡,如果我們想要給column加一個prefix惹谐, 可以用data.get_dummies里的prefix參數(shù)來實現(xiàn):
In [111]: dummies = pd.get_dummies(df['key'], prefix='key')
In [112]: df_with_dummy = df[['data1']].join(dummies)
In [113]: df_with_dummy
Out[113]:
data1 key_a key_b key_c
0 0 0 1 0
1 1 0 1 0
2 2 1 0 0
3 3 0 0 1
4 4 1 0 0
5 5 0 1 0
如果DataFrame中的某行同屬于多個分類狰住,則事情就會有點復(fù)雜墅垮。看一下MovieLens 1M數(shù)據(jù)集廓鞠,14章會更深入地研究它:
In [114]: mnames = ['movie_id', 'title', 'genres']
In [115]: movies = pd.read_table('datasets/movielens/movies.dat', sep='::',
.....: header=None, names=mnames)
In [116]: movies[:10]
Out[116]:
movie_id title genres
0 1 Toy Story (1995) Animation|Children's|Comedy
1 2 Jumanji (1995) Adventure|Children's|Fantasy
2 3 Grumpier Old Men (1995) Comedy|Romance
3 4 Waiting to Exhale (1995) Comedy|Drama
4 5 Father of the Bride Part II (1995) Comedy
5 6 Heat (1995) Action|Crime|Thriller
6 7 Sabrina (1995) Comedy|Romance
7 8 Tom and Huck (1995) Adventure|Children's
8 9 Sudden Death (1995)
Action
9 10 GoldenEye (1995) Action|Adventure|Thriller