0.需要的知識(shí)點(diǎn)
- 正則表達(dá)式
- java多線程線程池池知識(shí)
- httpclient網(wǎng)絡(luò)庫及json和html結(jié)構(gòu)
1.獲取主話題
在知乎中一共有33個(gè)主話題父款，在33個(gè)主話題下又有15776個(gè)子話題，因此我們首先要獲取到33個(gè)主話題
(ps:一開始打算用HttpURLConnection進(jìn)行網(wǎng)絡(luò)請(qǐng)求榴嗅，由于后面需要以post形式訪問并且提交form因此后面的代碼改用了httpclient進(jìn)行網(wǎng)絡(luò)請(qǐng)求)
圖中的data-id就是主話題的id
public static void getTopicId(){
new Thread(new Runnable() {
@Override
public void run() {
Connection connection;
int id = 1;
try {
URL url = new URL("https://www.zhihu.com/topics");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.connect();
BufferedReader bfr = new BufferedReader(new InputStreamReader(conn.getInputStream(),"UTF-8"));
String line = null;
StringBuilder sb = new StringBuilder();
while ((line = (bfr.readLine())) != null){
sb.append(line);
}
String result = sb.toString();
String regex = "data-id=\"[0-9]{0,6}\"";
Pattern pattern = Pattern.compile(regex);
Matcher m = pattern.matcher(result);
String regx = "href=\"#.*?\"";
Pattern p = Pattern.compile(regx);
Matcher mn = p.matcher(result);
while (m.find() && mn.find()){
String s = m.group();
s = s.substring(9,s.length() - 1);
String sn = mn.group();
sn = sn.substring(7,sn.length() - 1);
System.out.println(s + " " + sn);
connection = JDBCUtil.getConn();
PreparedStatement state = (PreparedStatement) connection.prepareStatement("insert into main_topic values(?,?,?)");
state.setInt(1, id++);
state.setInt(2, Integer.valueOf(s));
state.setString(3, sn);
state.execute();
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
}
2.獲取各個(gè)主話題下的分話題
通過抓包我們可以發(fā)現(xiàn)在分話題頁面通過post請(qǐng)求訪問到服務(wù)器來加載分話題，沒一次下拉到底部會(huì)多加載20個(gè)分話題，通過offset來控制熬粗，因此我們可以通過post請(qǐng)求服務(wù)器來獲得json數(shù)據(jù)
post請(qǐng)求結(jié)構(gòu)
通過postman模擬post請(qǐng)求得到返回?cái)?shù)據(jù)
由于返回的json數(shù)據(jù)中的中文為unicode編碼，因此運(yùn)用了Utils類中的方法來轉(zhuǎn)換為中文以便儲(chǔ)存到數(shù)據(jù)庫中
public static void getChildTopics(int topic_id,Connection conn) throws Exception{
int offset = 0;;
while (true){
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
.setRetryHandler(new DefaultHttpRequestRetryHandler())
.setConnectionManager(cm)
.build();
HttpPost req = new HttpPost("https://www.zhihu.com/node/TopicsPlazzaListV2");
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("method", "next"));
params.add(new BasicNameValuePair("params", "{\"topic_id\":" + topic_id + ",\"offset\":" + offset +",\"hash_id\":\"37492588249aa9b50ee49d1797e9af81\"}"));
req.setEntity(new UrlEncodedFormEntity(params,Consts.UTF_8));
HttpResponse resp = httpClient.execute(req);
String sb = EntityUtils.toString(resp.getEntity());
if (sb.length() < 25) break;
String regex = "<strong>.*?<\\\\/strong>";
Pattern p = Pattern.compile(regex);
Matcher m = p.matcher(sb);
String regx = "href=\\\\\"\\\\/topic\\\\/[0-9]+\\\\\"";
Pattern p1 = Pattern.compile(regx);
Matcher m1 = p1.matcher(sb);
while (m.find() && m1.find()){
String temp = m.group().substring(8, m.group().length() - 10);
String sql = "insert into child_topic values(null,?,?,?)";
PreparedStatement state = (PreparedStatement) conn.prepareStatement(sql);
state.setInt(1,topic_id);
state.setString(2,m1.group().substring(16, m1.group().length() - 2));
state.setString(3,Utils.decodeUnicode(temp));
state.execute();
}
offset += 20;
}
}
通過主話題id來獲取下面的子話題id
3.通過分話題id來獲取熱門問答
public static void loadOneTopicHotQA(int child_topic_id) throws Exception{
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
CloseableHttpClient httpClient = HttpClients.custom()
.setRetryHandler(new DefaultHttpRequestRetryHandler())
.setConnectionManager(cm)
.build();
HttpGet req = new HttpGet("https://www.zhihu.com/topic/" + child_topic_id + "/hot");
HttpResponse resp = httpClient.execute(req);
String result = EntityUtils.toString(resp.getEntity());
//正則匹配打不出來余境，后面以截圖形式給出
Pattern p_qa_id = Pattern.compile(regex_qa_id);
Pattern p_qa_name = Pattern.compile(regex_qa_name);
Pattern p_qa_username = Pattern.compile(regex_qa_username);
Matcher m_qa_id = p_qa_id.matcher(result);
Matcher m_qa_name = p_qa_name.matcher(result);
Matcher m_qa_username = p_qa_username.matcher(result);
while (m_qa_id.find() && m_qa_name.find() && m_qa_username.find()){
String[] qanda_id = m_qa_id.group().split("/");
String q_id = qanda_id[2];
String a_id = qanda_id[4].substring(0, qanda_id[4].length() - 2);
String q_name = m_qa_name.group().split("\n")[1];
String temp = m_qa_username.group();
String q_username = null;
if (temp.contains("匿名用戶")) q_username = "匿名用戶";
else if (temp.contains("知乎用戶")) q_username = "知乎用戶";
else q_username = temp.substring(30, temp.length() - 1);
HotQA qa = new HotQA(child_topic_id,Integer.valueOf(q_id), q_name, Integer.valueOf(a_id), q_username);
DaoImpl daoimpl = new DaoImpl();
daoimpl.save(qa, child_topic_id);
}
}
正則匹配
由于未登陸情況下可能會(huì)出現(xiàn)答主名字被替換為知乎用戶驻呐，因此要考慮答主名字為知乎用戶這種情況
4.爬到的數(shù)據(jù)能做什么
數(shù)據(jù)
- 通過child_topic_id可知道問題屬于哪個(gè)話題
- 通過question_id可以通過拼接url:https://www.zhihu.com/question/{question_id}來訪問問題
- 同理可得url:https://www.zhihu.com/question/{question_id}/answer/{answer_id}來訪問回答
- 同理:https://www.zhihu.com/people/{username}/answers
5.擴(kuò)展
在DAO層中提供了返回指定話題最新動(dòng)態(tài)的方法，同時(shí)也包含了通過answer_id獲取指定回答的方法芳来，將來可能還會(huì)加入爬取用戶信息的功能含末。
6.感謝
感謝看到這里，如果不嫌棄的話給gayhub項(xiàng)目一個(gè)star唄即舌，關(guān)注一下我也是最最最最吼的
gayhub項(xiàng)目 可以直接點(diǎn)喲
另外大家可以也可以通過微博聯(lián)系我喲：關(guān)耳金名
如何使用及編程過程中踩過的坑
