1. 引言
現(xiàn)代網(wǎng)頁往往其HTML只有基本結(jié)構(gòu)，而數(shù)據(jù)是通過AJAX或其他方法獲取后填充，這樣的模式對(duì)爬蟲有一定阻礙避诽，但是熟練以后獲取并不困難茉贡，本文以爬取天貓?jiān)u論為例簡單講講動(dòng)態(tài)獲取以及自定義Pipeline進(jìn)行數(shù)據(jù)清洗的過程。
2. 爬取商品信息
我們訪問s.taobao.com/search?q=你搜索的關(guān)鍵字 時(shí)可以很容易的獲取到搜索結(jié)果頁面，不難發(fā)現(xiàn)淘寶把搜索結(jié)果的信息嵌入到了該獲取結(jié)果的head標(biāo)簽之中，可以很容易的通過xpath將該信息抽取出來并整理成一個(gè)Json，你可以發(fā)現(xiàn)其中中文部分是由Unicode編碼編寫的帚豪，你可以自己寫一個(gè)convert函數(shù)去解決這個(gè)問題。
這里我也提供一個(gè)簡單的convert函數(shù)草丧，可以僅轉(zhuǎn)換文本中的Unicode編碼：
public class Unicode2utf8Utils {
public static String convert(String unicodeString) {
StringBuilder stringBuilder = new StringBuilder();
int i = -1;
int pos = 0;
while ((i = unicodeString.indexOf("\\u", pos)) != -1) {
stringBuilder.append(unicodeString.substring(pos, i));
if (i + 5 < unicodeString.length()) {
pos = i + 6;
stringBuilder.append((char) Integer.parseInt(unicodeString.substring(i + 2, i + 6), 16));
}
}
return stringBuilder.toString();
}
}
這里由于淘寶該數(shù)據(jù)是用Json格式發(fā)送的狸臣，可以很容易的用JsonPathSelector這個(gè)工具去獲取想要的字符，當(dāng)然昌执，使用前需要對(duì)數(shù)據(jù)進(jìn)行一些清理烛亦，讓它是一個(gè)方便識(shí)別的Json文本。（這些清理是根據(jù)具體實(shí)踐中爬到的內(nèi)容進(jìn)行分析得到的懂拾，你需要自己實(shí)踐看看需要清除哪些內(nèi)容煤禽。）
else if (page.getUrl().regex(urlList).match()) {
//獲取頁面script并轉(zhuǎn)碼為中文
String origin = Unicode2utf8Utils.convert(page.getHtml().xpath("http://head/script[7]").toString());
//從script中獲取json
Matcher jsonMatcher = Pattern.compile("\\{.*\\}").matcher(origin);
//如果成功獲取json數(shù)據(jù)
if (jsonMatcher.find()) {
//清理亂碼
String jsonString = jsonMatcher.group().replaceAll("\"navEntries\".*?,", "")
.replaceAll(",\"p4pdata\".*?\\\"\\}\"", "").replaceAll("\"spuList\".*?,", "");
//選擇auctions列表
List<String> auctions = new JsonPathSelector("mods.itemlist.data.auctions[*]").selectList(jsonString);
有關(guān)于JsonPathSelector的語法（JsonPath），可以參考這里JSONPath - XPath for JSON
使用方法即是新建一個(gè)JsonPathSelector并使用其select或selectList方法獲取所求元素岖赋。
對(duì)于每條元素的處理檬果，這里建議用阿里巴巴提供的fastjackson庫去操作。
//對(duì)于每一項(xiàng)商品
for (String auction : auctions) {
Map map = JSON.parseObject(auction);
//獲取評(píng)論url
String commentUrl = (String) map.get("comment_url");
if (commentUrl == null) continue;
//獲取商品id
Matcher itemIdMatcher = Pattern.compile("id=\\d+").matcher(commentUrl);
String itemIdString = null;
if (itemIdMatcher.find()) itemIdString = itemIdMatcher.group().replace("id=", "");
else continue;
//獲取商店ip
String shopLink = new JsonPathSelector("shopLink").select(auction);
Matcher shopIdMatcher = Pattern.compile("user_number_id=\\d+").matcher(shopLink);
String shopIdString = null;
if (shopIdMatcher.find()) shopIdString = shopIdMatcher.group().replace("user_number_id=", "");
else continue;
//記錄信息
map.put("itemId", itemIdString);
map.put("sellerId", shopIdString);
page.putField(itemIdString, map);
3. 爬取商品評(píng)論
至此我們的爬蟲已經(jīng)可以獲取每個(gè)關(guān)鍵詞的第一頁商品信息，那么选脊，如何獲取其評(píng)論呢杭抠？注意到我在上段代碼中獲取了itemId和sellerId，利用這兩個(gè)信息恳啥，我們可以獲取評(píng)論偏灿。
打開一個(gè)評(píng)論頁面，用瀏覽器的調(diào)試工具進(jìn)行查看角寸，會(huì)發(fā)現(xiàn)Network頁面中的各種請求菩混，我們可以挨個(gè)排查，查到扁藕，對(duì)于淘寶評(píng)論，將會(huì)發(fā)送如下請求：
https://rate.taobao.com/feedRateList.htm?auctionNumId=551058447857&userNumId=3167078258¤tPageNum=1&pageSize=20&rateType=&orderType=sort_weight&attribute=&sku=&hasSku=false&folded=0&ua=094%23UVQ6qM6U6l36u6ty666666BojjfaWoDLGsIU6Sf5RfSra4LjKWEeohUmxRUbiVNjH6Q6tusO%2Fbxm6M6QjLTM%2BR4t66W6nSkS1aQ6tHI6a486atAt6tlORWGWZHnBps8hHD80ee8tloiOPTTML6QtKBSv%2B6n0%2FxnKTeCbb1gD%2BlJiqwmPHGbgsSPNHG%2Fs%2FzmRR4pkHLd0%2BNJ9fpEJ%2FD76rYHg9yg9INGuG3hVxw01f9A2qP0vzP16Jjblbb%2FxxNnBuAmVHHEes9Jvkohr1G03Sv0yDg%2FAb9bnGzTspoo9%2B2raJp00HTElcncOzLSPIzvjT9n9zyoza5a2V7L%2BHpZYCWLYD7m%2F4est8Rws41d1V2R2D1jxDbS7Cn8Ez7C9w3FR3RoiAo0VcxtIsPgvI7SQVEjh9HS1bepYoRZep8Hws5zeVgc%2BApH6k0jKpgPs%2FzsBLuDczud0%2BNaAagcbpbHNvVTWALd414oEy3hV47Pv%2BU6Aa1ce1PZgkjc62Ty1ex77LRFAwLAk6M64jLTM%2B5PyzTXNAeTI09%2F0PkZvfRCGC15qjLTXi5Pyz9fNAehU09%2F0CRut66lLAeoM%2FDyr6M6ujLTWvMon0CR%3D&_ksTS=1498362441431_2073&callback=jsonp_tbcrate_reviews_list
對(duì)于天貓?jiān)u論疚脐，我們可以發(fā)現(xiàn)如下請求：
https://rate.tmall.com/list_detail_rate.htm?itemId=549440936281&spuId=846223934&sellerId=1996270577&order=3¤tPage=1&append=0&content=1&tagId=&posi=&picture=&ua=096UW5TcyMNYQwiAiwQRHhBfEF8QXtHcklnMWc%3D%7CUm5Ockt%2FSnRPcEh0T3pCfCo%3D%7CU2xMHDJ7G2AHYg8hAS8XIw0tA18%2BWDRTLVd5L3k%3D%7CVGhXd1llXGhdY1hnX2NYbVVrXGFDf0tyT3FJdEF8RHBNcU1zSnRaDA%3D%3D%7CVWldfS0SMg02Dy8QMB4jHzFnMQ%3D%3D%7CVmhIGCUFOBgkGiIePgc6BzsbJxkiFzcDPwAgHCIZLAw5AzxqPA%3D%3D%7CV2xMHDJXLwEhHSIcPAEhHSMeJHIk%7CWGFBET8RMQo%2BBiYdKBAwCz4GPmg%2B%7CWWBAED4QMAgxCioWKREtDTcPMApcCg%3D%3D%7CWmNDEz0TMwoxCSkVKhQvDzUBNQBWAA%3D%3D%7CW2NDEz0TM2NaZVx8QH9Dfl5gWmBAfkN8XmJWblBsU2tLd0p%2FX2NbDS0QMB4wECUcIRxKHA%3D%3D%7CXGVYZUV4WGdHe0J%2BXmBYYkJ7W2VYeExsWXlDY19nMQ%3D%3D&isg=AoKCefv4bxJaWnPIzoTSiRgq04hIRrnyMb30i8ybA_WoHyOZtOJ-fD4dtS2Y&needFold=0&_ksTS=1498362526461_1756&callback=jsonp1757
分析這兩個(gè)URL亿柑，我們可以發(fā)現(xiàn)，對(duì)于天貓連接棍弄，必要的屬性為itemId望薄，sellerId和currentPage，前兩個(gè)可以從商品信息的comment_url和shopLink兩個(gè)屬性中通過正則匹配獲取到呼畸，最后一個(gè)是頁數(shù)痕支。淘寶的連接也十分類似，只是屬性名稱有所變動(dòng)蛮原。因此卧须，我們可以針對(duì)這兩種連接發(fā)送AJAX請求。自己重新用這些屬性構(gòu)造連接后儒陨，能夠成功獲得評(píng)論信息花嘶。
if (!map.get("comment_count").toString().isEmpty()) {
if(commentUrl.contains("taobao")){
for (int i = 1; i <= 5; ++i) {
String taoBaoUrl = "https://rate.taobao.com/feedRateList.htm?auctionNumId=" + itemIdString + "&userNumId=" + shopIdString + "¤tPageNum=" + i;
page.addTargetRequest(taoBaoUrl);
}
}else {
for (int i = 1; i <= 5; ++i) {
String tmallUrl = "https://rate.tmall.com/list_detail_rate.htm?itemId=" + itemIdString + "&sellerId=" + shopIdString + "¤tPage=" + i;
page.addTargetRequest(tmallUrl);
}
}
}
對(duì)于評(píng)論信息的獲取如下：
if (page.getUrl().regex(tmallComment).match()) {
String text = page.getRawText().replace("\"rateDetail\":", "");
//記錄信息
Map map = JSON.parseObject(text);
if (map.get("rateList") == null) return;
Matcher itemIdMatcher = Pattern.compile("itemId=\\d+").matcher(page.getRequest().getUrl());
String itemIdString = null;
if (itemIdMatcher.find()) itemIdString = itemIdMatcher.group().replace("itemId=", "");
Matcher shopIdMatcher = Pattern.compile("sellerId=\\d+").matcher(page.getRequest().getUrl());
String shopIdString = null;
if (shopIdMatcher.find()) shopIdString = shopIdMatcher.group().replace("sellerId=", "");
Matcher currentPageMatcher = Pattern.compile("currentPage=\\d+").matcher(page.getRequest().getUrl());
String currentPageString = null;
if (currentPageMatcher.find()) currentPageString = currentPageMatcher.group().replace("currentPage=", "");
map.put("currentPage",currentPageString);
map.put("itemId", itemIdString);
map.put("sellerId", shopIdString);
map.put("url", page.getRequest().getUrl());
page.putField(itemIdString, map);
} else if (page.getUrl().regex(tbComment).match()) {
Matcher jsonMatcher = Pattern.compile("\\{.*\\}").matcher(page.getRawText());
if (jsonMatcher.find()) {
Map map = JSON.parseObject(jsonMatcher.group());
//如果觸發(fā)反爬蟲，報(bào)錯(cuò)
if (map.get("url") != null && map.get("url").toString().matches(urlSec)) {
System.out.println("Meet the anti-Spider!");
return;
}
if (map.get("comments") == null) return;
Matcher itemIdMatcher = Pattern.compile("auctionNumId=\\d+").matcher(page.getRequest().getUrl());
String itemIdString = null;
if (itemIdMatcher.find()) itemIdString = itemIdMatcher.group().replace("auctionNumId=", "");
Matcher shopIdMatcher = Pattern.compile("userNumId=\\d+").matcher(page.getRequest().getUrl());
String shopIdString = null;
if (shopIdMatcher.find()) shopIdString = shopIdMatcher.group().replace("userNumId=", "");
Matcher currentPageMatcher = Pattern.compile("currentPageNum=\\d+").matcher(page.getRequest().getUrl());
String currentPageString = null;
if (currentPageMatcher.find()) currentPageString = currentPageMatcher.group().replace("currentPageNum=", "");
map.put("currentPage",currentPageString);
map.put("itemId", itemIdString);
map.put("sellerId", shopIdString);
map.put("url", page.getRequest().getUrl());
page.putField(itemIdString, map);
}
}
這里可以看到蹦漠，處理淘寶評(píng)論和天貓?jiān)u論的過程是十分相似的椭员，但是天貓?jiān)u論沒有反爬，而淘寶評(píng)論會(huì)有反爬手段笛园，目前我的一些簡單的規(guī)避反爬蟲的方法都不奏效隘击，也未能推測出反爬的方法，因此這個(gè)爬蟲幾乎只能獲取一條商品一頁的淘寶評(píng)論研铆。想獲取更多埋同，還要想出躲避反爬的方法。因此我標(biāo)題里僅說天貓?jiān)u論蚜印。
4. 數(shù)據(jù)清洗
webmagic的Pipeline是可以自定義的莺禁，因而可以在其中進(jìn)行數(shù)據(jù)清洗工作，以使數(shù)據(jù)在爬取后自動(dòng)清洗窄赋。這里給出我自定義的Pipeline：
public class MyTBJsonPipeline extends FilePersistentBase implements Pipeline {
public MyTBJsonPipeline(String path) {
this.setPath(path);
}
@Override
public void process(ResultItems resultItems, Task task) {
try {
Iterator iterator = resultItems.getAll().values().iterator();
while (iterator.hasNext()) {
Map map = (Map) iterator.next();
String name = map.get("itemId").toString();
if (map.get("raw_title") == null) {
if (map.get("rateList")!=null)
name += "_tmall_comment";
else name += "_taobao_comment";
name+="_"+map.get("currentPage");
}
PrintWriter printWriter = new PrintWriter(new FileWriter(this.getFile(path + name + ".json")));
printWriter.write(JSON.toJSONString(map));
printWriter.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
我針對(duì)每一條商品都存儲(chǔ)了單獨(dú)的文件哟冬，并將評(píng)論單獨(dú)存儲(chǔ)（文件名有所關(guān)聯(lián)）楼熄，進(jìn)而能夠清晰的展示信息。
完整代碼見個(gè)人github：https://github.com/CieloSun/FashionSpider
