1. 水文數(shù)據(jù)集
https://data.edmonton.ca/dataset/Water-Levels-and-Flows/cnsu-iagr
###該數(shù)據(jù)集一共9個(gè)屬性:
1.Raw ID
2.Station Number => 水文站編號 --26 --離散值
3.Station Description => 水文站描述 --26 --文本信息
4.Latitude => 緯度 --28 --離散值
5.Longitude => 經(jīng)度 --28 --離散值
6.Station Location => 水文站位置乐疆，由經(jīng)緯度組合而成 --28 --離散值
7.Date and Time => 時(shí)間
8.Water Level (m) => 水位 -- --連續(xù)值
9.Water Flow (cu meter per sec) => 流速 -- --連續(xù)值
因?yàn)門POT只能處理數(shù)值類型焰望，在我們的數(shù)據(jù)集中蛋勺，有4個(gè)變量不是數(shù)值類型:
Station Number, Station Description, Station Location, Date and Time
所以侦讨，下面重點(diǎn)是處理這幾個(gè)變量摧找；
###數(shù)據(jù)預(yù)處理:
1.Station Number
離散值养交，有26個(gè)可選抓狭，使用one-hot編碼
2.Station Description， Station Location, Raw ID
無用信息粱挡，刪除
3.Date and Time
分析數(shù)據(jù)赠幕，從2016年到2019年，所以年份不敏感询筏；
拆分為月榕堰，日和小時(shí)
4.Latitude和Longitude可能缺失
總的記錄數(shù):
(3910803, 9)
非空記錄數(shù):
(1975617, 9)
5.經(jīng)緯度缺失的數(shù)據(jù)量很大，所以不能直接舍棄嫌套；
首先計(jì)算水文站經(jīng)緯度的平均值逆屡，用這個(gè)來代替
6.Water Flow (cu meter per sec)可能非法，例如-1, 刪除這些記錄
2. 代碼
from tpot import TPOTRegressor
from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np
import os
from tensorflow.python.client import device_lib
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
print(device_lib.list_local_devices())
df = pd.read_csv('Water_Levels_and_Flows.csv.1w', header=0,error_bad_lines=False)
print(df.head(5))
print(df.shape)
print(pd.isnull(df).any())
#刪除Water_Flow為負(fù)的行
print('刪除Water_Flow無效行')
df=df[~(df['Water_Flow']<0)]
print(df.shape)
mean_longitude = df.groupby('Station Number').Longitude.mean()
mean_latitude = df.groupby('Station Number').Latitude.mean()
null_longitude = df[df.Longitude.isnull()]
for cat in ['Station Number', 'Station Description', 'Latitude', 'Longitude' ]:
print("Number of levels in category '{0}': \b {1:2.2f} ".format(cat, df[cat].unique().size))
for cat in ['Latitude', 'Longitude']:
print("Levels for catgeory '{0}': {1}".format(cat, df[cat].unique()))
#刪掉無關(guān)因素
df = df.drop(['id', 'Station Description','Station Location', 'Latitude', 'Longitude'],axis=1) # axis=0 刪除行踱讨，=1 刪除列
#將Station用啞變量表示
dumm = pd.get_dummies(df[['Station Number']]) # '啞變量'矩陣
df = df.join(dumm)
del df['Station Number'] # 刪除
#添加month,day,hour三個(gè)鍵和值
temp=pd.DatetimeIndex(df['Date and Time'])
df['date']=temp.date
df['time']=temp.time
df['hour']=pd.to_datetime(df.time,format="%H:%M:%S")#變換格式
df['hour']=pd.Index(df["hour"]).hour
df['month']=pd.to_datetime(df.date,format="%Y-%m-%d")#變換格式
df['month']=pd.Index(df["month"]).month
df['day']=pd.to_datetime(df.date,format="%Y-%m-%d")#變換格式
df['day']=pd.Index(df["day"]).day
df = df.drop(['Date and Time', 'date', 'time'],axis=1) # axis=0 刪除行魏蔗，=1 刪除列
print(df.head(5))
print(df.shape)
X = np.array(df.drop(['Water_Level'], 1))
Y = np.array(df['Water_Level'])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2)
tpot = TPOTRegressor(generations=20, verbosity=2) #迭代20次
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
tpot.export('pipeline.py')
3. 運(yùn)行TPOT
因?yàn)閳?zhí)行時(shí)間過長，所以使用后臺運(yùn)行方式:
nohup python -u water_tpot.py > nohup.out 2>&1 &
輸出如下:
Generation 1 - Current best internal CV score: -0.10354635942452688
Generation 2 - Current best internal CV score: -0.07407627082459693
Generation 3 - Current best internal CV score: -0.07407627082459693
Generation 4 - Current best internal CV score: -0.07407627082459693
Generation 5 - Current best internal CV score: -0.07261441673202419
Generation 6 - Current best internal CV score: -0.07261441673202419
Generation 7 - Current best internal CV score: -0.07261441673202419
Generation 8 - Current best internal CV score: -0.06272878450716013
Generation 9 - Current best internal CV score: -0.06150379726583473
Generation 10 - Current best internal CV score: -0.06150379726583473
Generation 11 - Current best internal CV score: -0.06150379726583473
Generation 12 - Current best internal CV score: -0.05930206668688394
Generation 13 - Current best internal CV score: -0.054799951152979184
Generation 14 - Current best internal CV score: -0.054799951152979184
Generation 15 - Current best internal CV score: -0.052015877476651276
Generation 16 - Current best internal CV score: -0.052015877476651276
Generation 17 - Current best internal CV score: -0.05173009225925466
Generation 18 - Current best internal CV score: -0.043718068024817355
Generation 19 - Current best internal CV score: -0.043718068024817355
Generation 20 - Current best internal CV score: -0.043718068024817355
Best pipeline: ExtraTreesRegressor(SelectPercentile(DecisionTreeRegressor(ElasticNetCV(input_matrix, l1_ratio=0.7000000000000001, tol=1e-05), max_depth=1, min_samples_leaf=2, min_samples_split=15), percentile=94), bootstrap=False, max_features=1.0, min_samples_leaf=2, min_samples_split=2, n_estimators=100)
-0.027241579610884296
TPOT認(rèn)為ExtraTreesRegressor是最好的模型痹筛，并導(dǎo)出pipeline.py莺治。
4. 運(yùn)行pipeline
導(dǎo)出的pipeline.py文件需要根據(jù)實(shí)際情況修改如下:
import numpy as np
import pandas as pd
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.feature_selection import SelectPercentile, f_regression
from sklearn.linear_model import ElasticNetCV
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, make_union
from sklearn.tree import DecisionTreeRegressor
from tpot.builtins import StackingEstimator
# NOTE: Make sure that the class is labeled 'target' in the data file
df = pd.read_csv('Water_Levels_and_Flows.csv.1w', sep=',', header=0,error_bad_lines=False)
#刪除Water_Flow為負(fù)的行
print('刪除Water_Flow無效行')
df=df[~(df['Water_Flow']<0)]
print(df.shape)
mean_longitude = df.groupby('Station Number').Longitude.mean()
mean_latitude = df.groupby('Station Number').Latitude.mean()
null_longitude = df[df.Longitude.isnull()]
for cat in ['Station Number', 'Station Description', 'Latitude', 'Longitude' ]:
print("Number of levels in category '{0}': \b {1:2.2f} ".format(cat, df[cat].unique().size))
for cat in ['Latitude', 'Longitude']:
print("Levels for catgeory '{0}': {1}".format(cat, df[cat].unique()))
#刪掉無關(guān)因素
df = df.drop(['id', 'Station Description','Station Location', 'Latitude', 'Longitude'],axis=1) # axis=0 刪除行，=1 刪除列
#將Station用啞變量表示
dumm = pd.get_dummies(df[['Station Number']]) # '啞變量'矩陣
df = df.join(dumm)
del df['Station Number'] # 刪除
#添加month,day,hour三個(gè)鍵和值
temp=pd.DatetimeIndex(df['Date and Time'])
df['date']=temp.date
df['time']=temp.time
df['hour']=pd.to_datetime(df.time,format="%H:%M:%S")#變換格式
df['hour']=pd.Index(df["hour"]).hour
df['month']=pd.to_datetime(df.date,format="%Y-%m-%d")#變換格式
df['month']=pd.Index(df["month"]).month
df['day']=pd.to_datetime(df.date,format="%Y-%m-%d")#變換格式
df['day']=pd.Index(df["day"]).day
df = df.drop(['Date and Time', 'date', 'time'],axis=1) # axis=0 刪除行帚稠，=1 刪除列
print(df.head(5))
print(df.shape)
X = np.array(df.drop(['Water_Level'], 1))
Y = np.array(df['Water_Level'])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2)
training_features, testing_features, training_target, testing_target = \
train_test_split(X, Y, random_state=None)
# Average CV score on the training set was:-0.043718068024817355
exported_pipeline = make_pipeline(
StackingEstimator(estimator=ElasticNetCV(l1_ratio=0.7000000000000001, tol=1e-05)),
StackingEstimator(estimator=DecisionTreeRegressor(max_depth=1, min_samples_leaf=2, min_samples_split=15)),
SelectPercentile(score_func=f_regression, percentile=94),
ExtraTreesRegressor(bootstrap=False, max_features=1.0, min_samples_leaf=2, min_samples_split=2, n_estimators=100)
)
exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)
print("results: %s"%results)
score = exported_pipeline.score(testing_features, testing_target)
print("score: %s"%score)
同樣谣旁，在后臺運(yùn)行:
nohup python -u pipeline.py > nohup.out 2>&1 &
輸出如下:
results: [7.1012806 1.25462311 0.73743519 ... 2.12231535 4.22561277 1.27338528]
score: 0.9999991261258582
5. GPU
可惜的是，目前TPOT不支持GPU：
https://github.com/EpistasisLab/tpot/issues/680
不過滋早，實(shí)際應(yīng)用中我們發(fā)現(xiàn)榄审，只是GPU-Util為0， GPU Memory Usage仍然可以達(dá)到10440MiB
屏幕快照 2019-01-30 下午6.32.55.png
6. 速度
TPOT在處理小規(guī)模數(shù)據(jù)的時(shí)候非掣唆铮快搁进，結(jié)果很給力。但處理大規(guī)模的數(shù)據(jù)問題昔头，速度非常慢饼问。所以可以嘗試在數(shù)據(jù)清洗之后，抽樣小部分?jǐn)?shù)據(jù)跑一下TPOT揭斧，能得到一個(gè)還不錯(cuò)的算法匆瓜。
