本文主要記錄利用爬蟲爬取豆瓣對電影《西虹市首富》的短評，使用word分詞器分詞萝快，并使用Spark計算出磁盤取Top20尚粘，使用echats展示。

效果圖如下：

相關(guān)文章：
1.Spark之PI本地
2.Spark之WordCount集群
3.SparkStreaming之讀取Kafka數(shù)據(jù)
4.SparkStreaming之使用redis保存Kafka的Offset
5.SparkStreaming之優(yōu)雅停止
6.SparkStreaming之寫數(shù)據(jù)到Kafka
7.Spark計算《西虹市首富》短評詞云

1.爬取數(shù)據(jù)

參考：使用爬蟲爬取豆瓣電影影評數(shù)據(jù)Java版

其中略微修改：
PageParser.java

public Data<T> parse(String url, String html) {

  Document doc = Jsoup.parse(html, url);

  // 獲取鏈接列表
  List<String> links =
    doc.select("#paginator > a.next")
    .stream()
    .map(a -> a.attr("abs:href"))
    .collect(Collectors.toList());

  // 獲取數(shù)據(jù)列表
  List<Map<String, Object>> results = doc.select("#comments > div.comment-item")
    .stream()
    .map(div -> {
      Map<String, Object> data = new HashMap<>();

      String author = div.selectFirst("h3 > span.comment-info > a").text();
      String date = div.selectFirst("h3 > span.comment-info > span.comment-time").text();
      Element rating = div.selectFirst("h3 > span.comment-info > span.rating");
      String star = "0";
      if (rating != null) {
        // allstar40 rating
        star = rating.attr("class");
        star = star.substring(7, 9);
      }
      String vote = div.selectFirst("h3 > span.comment-vote > span.votes").text();
      String comment = div.selectFirst("div.comment > p").text();

      data.put("author", author);
      data.put("date", date);
      if (star != null)
        data.put("star", star);
      data.put("vote", vote);
      data.put("comment", comment);

      return data;
    })
    .collect(Collectors.toList());

  return new Data(links, results);
}

DataProcessor.java

public void process(List<T> results) {
  if (results == null || results.isEmpty()) {
    return;
  }

  try {

    // 數(shù)據(jù)
    BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File("C:\\xhs_json.txt"), true)));
    Gson gson = new Gson();
    for (T result : results) {
      bw.write(gson.toJson(result));
      bw.write("\r\n");
    }
    bw.flush();
    bw.close();

    // 分詞結(jié)果
    PrintWriter pw = new PrintWriter(new OutputStreamWriter(new FileOutputStream(new File("C:\\xhs_word.txt"), true)));
    for (T result : results) {
      if (result instanceof Map) {
        List<Word> words = WordSegmenter.seg(((Map) result).get("comment").toString());
        pw.println(words.stream().map(word -> word.getText()).collect(Collectors.joining(" ")));
      }
    }
    pw.flush();
    pw.close();
  } catch (Exception e) {
    e.printStackTrace();
  }

}

大概540條數(shù)據(jù)衫仑，保存兩份文件，xhs_json.txt是完整的短評json文件堕花，xhs_word.txt是使用word對短評內(nèi)容分詞的文件

xhs_json.txt

xhs_word.txt

爬蟲下載地址
xhs_json.txt下載地址
xhs_word.txt下載地址

2.Spark計算

只需要利用xhs_word.txt文件進(jìn)行wordcount計算即可文狱，然后打印出echat需要顯示的格式即可

object YingPing {
  def main(args: Array[String]): Unit = {
    //創(chuàng)建一個Config
    val conf = new SparkConf()
      .setAppName("YingPing")
      .setMaster("local[1]")

    //核心創(chuàng)建SparkContext對象
    val sc = new SparkContext(conf)

    //WordCount
    sc.textFile("C:\\xhs_word.txt")
      .flatMap(_.split(" "))
      .map((_, 1))
      .reduceByKey(_ + _)
      //.repartition(1)
      .sortBy(_._2, false)
      .take(20)
      .map(x => {
        val map = new java.util.HashMap[String, String]()
        map.put("name", x._1)
        map.put("value", x._2 + "")
        map.put("itemStyle", "createRandomItemStyle()")
        map
      })
      .foreach(item => println(new Gson().toJson(item).replace("\"c", "c").replace(")\"", ")") + ","))
    // 借助http://echarts.baidu.com/echarts2/doc/example/wordCloud.html#infographic可以顯示詞云

    //停止SparkContext對象
    sc.stop()
  }
}

結(jié)果如下：

{"name":"電影","itemStyle":createRandomItemStyle(),"value":"160"},
{"name":"麻花","itemStyle":createRandomItemStyle(),"value":"112"},
{"name":"喜劇","itemStyle":createRandomItemStyle(),"value":"100"},
{"name":"開心","itemStyle":createRandomItemStyle(),"value":"96"},
{"name":"沈騰","itemStyle":createRandomItemStyle(),"value":"92"},
{"name":"笑點(diǎn)","itemStyle":createRandomItemStyle(),"value":"92"},
{"name":"笑","itemStyle":createRandomItemStyle(),"value":"79"},
{"name":"真的","itemStyle":createRandomItemStyle(),"value":"50"},
{"name":"好笑","itemStyle":createRandomItemStyle(),"value":"49"},
{"name":"一部","itemStyle":createRandomItemStyle(),"value":"47"},
{"name":"故事","itemStyle":createRandomItemStyle(),"value":"47"},
{"name":"諷刺","itemStyle":createRandomItemStyle(),"value":"45"},
{"name":"太","itemStyle":createRandomItemStyle(),"value":"44"},
{"name":"尷尬","itemStyle":createRandomItemStyle(),"value":"40"},
{"name":"星","itemStyle":createRandomItemStyle(),"value":"39"},
{"name":"尬","itemStyle":createRandomItemStyle(),"value":"37"},
{"name":"夏洛特","itemStyle":createRandomItemStyle(),"value":"34"},
{"name":"觀眾","itemStyle":createRandomItemStyle(),"value":"33"},
{"name":"金錢","itemStyle":createRandomItemStyle(),"value":"33"},
{"name":"挺","itemStyle":createRandomItemStyle(),"value":"33"},

將結(jié)果復(fù)制到echats的在線頁面顯示即可

image.png