前言：
對(duì)于像學(xué)大數(shù)據(jù)泉瞻，機(jī)器學(xué)習(xí)种玛，深度學(xué)習(xí)相關(guān)的初學(xué)者往往面臨著兩個(gè)問題：
- 沒有合適的訓(xùn)練數(shù)據(jù)
- 機(jī)器不行，沒有集群
這里推薦一個(gè)不錯(cuò)的平臺(tái)，相信大家都聽過鳞仙，就是kaggle平臺(tái)了，這里提供不少數(shù)據(jù)的的下載，和相關(guān)比賽。
View
基本是圍繞著competition就是比賽進(jìn)行的垮兑。
打開比賽我們可以看到很多，注意下左面的有顏色的條漱挎，綠色是簡(jiǎn)單的系枪，紅色是難，還有藍(lán)色磕谅，橙色等私爷。
我們作為初學(xué)者橙色以上的就不用看了。膊夹。衬浑。
Getting started
這里我選擇一個(gè)入門級(jí)別的題，Digit Recognizer放刨，就是數(shù)字識(shí)別工秩，點(diǎn)開后，我們點(diǎn)擊data进统，
Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.
可以看到描述是28 * 28的圖片數(shù)據(jù)助币，但是和tensoflow的那個(gè)MNISTdemo不同的是這份數(shù)據(jù)的格式是csv，每一張圖片的像素共 784個(gè)(28*28),矩陣展開成一維的螟碎，我們?cè)倏纯雌渌腸ompetition眉菱，基本上都是csv格式的數(shù)據(jù)，不同手動(dòng)分?jǐn)?shù)據(jù)抚芦，這里已經(jīng)給我們分開了test和train數(shù)據(jù)倍谜，我們要做的就是將最后的預(yù)測(cè)結(jié)果還是以csv的形式上傳即可，系統(tǒng)為我們?cè)u(píng)分叉抡。
需要注意的是尔崔，不上傳代碼，就是不論我們用什么算法褥民，框架季春，只要結(jié)果對(duì)就行，不管是spark mllib 消返，py sklearn载弄， tensorflow，caffe/caffe2都行~
給一個(gè)我實(shí)現(xiàn)的隨機(jī)森林分類樹的解法：
#!/usr/bin/python
# -*- coding: utf-8 -*-
from numpy import *
import csv
# array format to int
def toInt(array):
array=mat(array)
m,n=shape(array)
newmat=zeros((m,n))
for i in xrange(m):
for j in xrange(n):
newmat[i,j]=int(array[i,j])
return newmat
def nomalizing(array):
m,n=shape(array)
for i in xrange(m):
for j in xrange(n):
if array[i,j]!=0:
if array[i,j] > 128:
array[i,j] = 2
else:
array[i,j] = 1
return array
def loadTrainData():
l=[]
with open('train.csv','r') as fp:
lines=csv.reader(fp)
for line in lines:
l.append(line) #42001*785
#remove title
l.remove(l[0])
l=array(l)
label=l[:,0]
data=l[:,1:]
return nomalizing(toInt(data)),toInt(label) #label 1*42000 data 42000*784
def loadTestData():
l=[]
with open('test.csv') as file:
lines=csv.reader(file)
for line in lines:
l.append(line)#28001*784
l.remove(l[0])
data=array(l)
return nomalizing(toInt(data)) # data 28000*784
def saveResult(result,csvName):
with open(csvName,'wb') as myFile:
myWriter=csv.writer(myFile)
myWriter.writerow(["ImageId","Label"])
index=0;
for i in result:
tmp=[]
index=index+1
tmp.append(index)
#tmp.append(i)
tmp.append(int(i))
myWriter.writerow(tmp)
from sklearn.ensemble import RandomForestClassifier
def RFClassify(trainData,trainLabel,testData):
nbCF=RandomForestClassifier(n_estimators=200,warm_start = True)
nbCF.fit(trainData,ravel(trainLabel))
testLabel=nbCF.predict(testData)
saveResult(testLabel,'Result.csv')
return testLabel
def dRecognition():
trainData,trainLabel=loadTrainData()
print "load train data finish"
testData=loadTestData()
print "load test data finish"
result=RFClassify(trainData,trainLabel,testData)
print "finish!"
if __name__ == '__main__':
dRecognition()
開始跑撵颊，如果機(jī)器好的話宇攻，也就2-3min就跑完了，畢竟不是DNN(深度神經(jīng)網(wǎng)絡(luò))嘛
然后將得到的文件上傳倡勇。
可以看到我提交了4次（每次優(yōu)化下系數(shù)逞刷，比如樹的棵樹，是否是純凈葉子節(jié)點(diǎn)等。夸浅。）最好的accuracy rate是0.96771,之前我就明白仑最，什么都可以用隨機(jī)森林，但都不夠準(zhǔn)確帆喇。警医。
我們看看上面排名的：
厲害了，100%的都有坯钦，說明題目本身是可以獲得精確匹配的预皇。我們看不到人家的代碼，也就不知道用了什么方法葫笼，或者外星人深啤、、路星、額溯街，扯遠(yuǎn)了。
神經(jīng)網(wǎng)絡(luò)解法：
這里還有個(gè)比較好洋丐，就是提供了comment呈昔，我們打開一個(gè)，根據(jù)他的note我大概給出一個(gè)TF的版本友绝，但由于還在上班堤尾，我沒什么環(huán)境跑，（公司提供的tensoflow on Yarn迁客，我還沒研究明白郭宝。。掷漱。）還是回校后粘室，沒啥事的時(shí)候試試“單機(jī)單卡”吧，畢竟沒錢買好的GPU卜范，唉~~
代碼：
import numpy as np
import pandas as pd
import scipy
import tensorflow as tf
# The competition datafiles are in the directory ../input
# Read competition data files:
filedir = os.listdir()
train = pd.read_csv("/home/hdp-like/bigame_gbw/TF28_28/train.csv")
test = pd.read_csv("/home/hdp-like/bigame_gbw/TF28_28/test.csv")
# Write to the log:
print("Training set has {0[0]} rows and {0[1]} columns".format(train.shape))
print("Test set has {0[0]} rows and {0[1]} columns".format(test.shape))
learning_rate = 0.01
training_iteration = 30
batch_size = 300
display_step = 2
trainfv = train.drop(['label'], axis=1).values.astype(dtype=np.float32)
trainLabels = train['label'].tolist()
ohtrainLabels = tf.one_hot(trainLabels, depth=10)
ohtrainLabelsNdarray = tf.Session().run(ohtrainLabels).astype(dtype=np.float64)
trainfv = np.multiply(trainfv, 1.0 / 255.0)
testData = test.values
testData = np.multiply(testData, 1.0 / 255.0)
#train_x,train_y = make_the_data_ready_conv(train_x,train_y)
#valid_x,valid_y = make_the_data_ready_conv(valid_x,valid_y)
x = tf.placeholder("float",[None,784])
y = tf.placeholder("float",[None,10])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
with tf.name_scope("Wx_b") as scope:
model = tf.nn.softmax(tf.matmul(x,W) + b)
w_h = tf.summary.histogram("weights",W)
b_h = tf.summary.histogram("biases",b)
#loss function with maximum likelihood
with tf.name_scope("cost_function") as scope:
cost_function = -tf.reduce_sum(y * tf.log(model))
tf.summary.scalar("cost_function",cost_function)
#optimization with SGD
with tf.name_scope("train") as scope:
op = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost_function)
init = tf.initialize_all_variables()
merged_summary_op = tf.summary.merge_all()
import math
from random import randint
# for faster convergence
def random_batch(data,labels,size):
value = math.floor(len(data) / size)
intervall = randint(0,value-1)
return data[intervall*size:intervall*(size+1)],labels[intervall*size:intervall*(size+1)]
with tf.Session() as sess:
sess.run(init)
#run as single machine multiply GPU
# view the stuff on tensorboard
for iteration in range(training_iteration):
avg_cost = 0
total_batch = int(trainfv.shape[0]/batch_size)
for i in range(total_batch):
batch_xs,batch_ys = random_batch(trainfv,ohtrainLabelsNdarray,total_batch)
sess.run(op,feed_dict={x: batch_xs, y: batch_ys})
avg_cost += sess.run(cost_function,feed_dict={x: batch_xs, y: batch_ys}) / total_batch
if iteration % display_step == 0:
print "Iteration:", '%04d' % (iteration + 1), "cost=", "{:.9f}".format(avg_cost)
print "Training Finished"
correct_prediction = tf.equal(tf.argmax(model, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print "\nAccuracy of the current model: ",sess.run(accuracy, feed_dict={x: trainfv[0:10000], y: ohtrainLabelsNdarray[0:10000]})
prob = sess.run(tf.argmax(model,1), feed_dict = {x: testData})
which = 1
print 'predicted labe: {}'.format(str(prob[which]))
print(prob)
#import csv
#outputFile_dir = '../input/output.csv'
#header = ['ImageID','Label']
#with open(outputFile_dir, 'w', newline='') as csvFile:
# writer = csv.writer(csvFile, delimiter = ',')
# writer.writerow(header)
# for i, p in enumerate(prob):
# writer.writerow([str(i+1), str(p)])
哦衔统，還有就是注釋啥的別用中文，生成出來的文件是Utf8海雪，這可能影響檢查的準(zhǔn)確性锦爵。。奥裸。還有就是险掀，中文注釋多l(xiāng)ow呀~
結(jié)束了
之前有個(gè)大二的學(xué)弟or學(xué)妹（我不知道男女）問我，我想入門大數(shù)據(jù)湾宙，做了幾個(gè)featured的,幾個(gè)迷郑。枝恋。還是featured的，當(dāng)時(shí)我回復(fù)他微信嗡害，默默地看了看朋友圈，畦攘，霸妹，，人家把我屏蔽了知押√久看來是遇到大神了，要不就是外星人~我只求台盯，永遠(yuǎn)不拉黑我就好
