Android 使用Jsoup爬取碼云開源項目

先放一下官方文檔

implementation 'org.jsoup:jsoup:1.12.1'

Jsoup可以把網(wǎng)頁解析成Document對象，然后我們根據(jù)對應的元素id或者class以及其他的屬性当宴，獲取對應的信息

幾個重要且常用的方法：
　connect(url): 從URL獲取HTML來解析
　getElementById(String id)：通過id來獲取
　getElementsByTag(String tagName)：通過標簽名字來獲取
　getElementsByClass(String className)：通過類名來獲取
　getElementsByAttribute(String key)：通過屬性名字來獲取
　getElementsByAttributeValue(String key, String value)：通過指定的屬性名字，屬性值來獲取
　getAllElements()：獲取所有元素
　select(String cssQuery)：通過類似于css或jQuery的選擇器來查找元素

話不多說擦囊，直接上教程

進入碼云->移動開發(fā)

碼云.jpg

按F12查看網(wǎng)頁內容拦键，按shift+ctrl+C選中列表可以在右邊查看到對應的信息

dev_html.jpg

每個元素的屬性都清楚后，開始寫代碼解析
根據(jù)上面分析的信息先建個文章實體類

data class ArticleBean(val avatar: String, //頭像
                       val title: String,  //標題
                       val label: String,  //分類
                       val watchers: Int,  //關注數(shù)
                       val collect: Int,   //收藏數(shù)
                       val forkNum: Int,   //fork數(shù)
                       val desc: String,   //描述
                       val date: String,   //時間
                       val urlLink: String //文章跳轉鏈接
): Serializable

然后使用上面提到的connect方法去獲取document

val url = "https://gitee.com/explore/mobile-develop?order=recommend"
val document = Jsoup.connect(url).get()

獲取到網(wǎng)頁內容后開始解析它行贪，根據(jù)上面的截圖可以看到，列表的內容被一個div包裹起來了模闲，這個div的class是ui relaxed divided items explore-repo__list建瘫，那我們就根據(jù)這個class去獲取我們需要的文章，注意：在填寫class時要將中間的空格改成"."

val elements = document.select("div.ui.relaxed.divided.items.explore-repo__list").select("div.item")

如果說這個屬性沒有class围橡，但是有id暖混，這時候就應該將"."(不是上面說的空格)改為"#"

val elements = document.select("div#ui.relaxed.divided.items.explore-repo__list").select("div.item")

article_item.jpg

下面就是重復上面的步驟，根據(jù)class或者id翁授，獲取我們需要的數(shù)據(jù)拣播，一層層解析就好了

elements.forEach {
                val social = it.select("div.content").select("div.explore-project__meta-social.pull-right").select("a")
                articles.add(
                    ArticleBean(
                        it.select("a.project-creator-link.ui.avatar.image.pull-left").select("img").attr("src"),
                        it.select("div.content").select("a.title.project-namespace-path").text(),
                        it.select("div.content").select("span>a").text(),
                        social[0].text().toInt(),
                        social[1].text().toInt(),
                        social[2].text().toInt(),
                        it.select("div.project-desc").text(),
                        it.select("div.project-latest").text(),
                        baseUrl + it.select("a.project-creator-link.ui.avatar.image.pull-left").attr("href")
                ))
            }

Video_20190719_031346_446.gif

MainActivity:

class MainActivity : AppCompatActivity(), ArticleAdapter.OnItemClick {
    private var articles = ArrayList<ArticleBean>()
    private val mAdapter: ArticleAdapter by lazy {
        ArticleAdapter(articles, this)
    }

    private val baseUrl = "https://gitee.com"

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        recyclerView.run {
            mAdapter.setOnItenClick(this@MainActivity)
            adapter = mAdapter
            layoutManager = LinearLayoutManager(this@MainActivity)
            addItemDecoration(DividerItemDecoration(this@MainActivity, DividerItemDecoration.VERTICAL))
        }

        Thread(Runnable { jsoupData() }).start()
    }

    private fun jsoupData() {
        val url = "https://gitee.com/explore/mobile-develop?order=recommend"

        try {
            val document = Jsoup.connect(url).get()
            val elements = document.select("div.ui.relaxed.divided.items.explore-repo__list").select("div.item")
            elements.forEach {
                val social = it.select("div.content").select("div.explore-project__meta-social.pull-right").select("a")
                articles.add(
                    ArticleBean(
                        it.select("a.project-creator-link.ui.avatar.image.pull-left").select("img").attr("src"),
                        it.select("div.content").select("a.title.project-namespace-path").text(),
                        it.select("div.content").select("span>a").text(),
                        social[0].text().toInt(),
                        social[1].text().toInt(),
                        social[2].text().toInt(),
                        it.select("div.project-desc").text(),
                        it.select("div.project-latest").text(),
                        baseUrl + it.select("a.project-creator-link.ui.avatar.image.pull-left").attr("href")
                ))
            }
            runOnUiThread {
                mAdapter.notifyDataSetChanged()
            }
        } catch (e: Exception) {
            Log.e("error--->", e.message)
        }
    }

    override fun onClick(articleBean: ArticleBean, position: Int) {
        val intent = Intent(this, WebviewActivity::class.java)
        intent.putExtra("articleBean", articleBean)
        startActivity(intent)
    }
}

adapter：

class ArticleAdapter(var items: List<ArticleBean>, val content: Context): RecyclerView.Adapter<ArticleAdapter.ViewHolder>() {

    private var onItemClick: OnItemClick? = null

    public fun setOnItenClick(onItemClick: OnItemClick){
        this.onItemClick = onItemClick
    }

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewHolder {
        val view = LayoutInflater.from(parent.context).inflate(R.layout.article_item_layout, parent, false)
        val holder = ViewHolder(view)
        if (onItemClick != null){
            holder.itemView.setOnClickListener {
                val position = holder.adapterPosition
                val articleBean = items.get(position)
                onItemClick?.onClick(articleBean, position)
            }
        }
        return holder
    }

    override fun getItemCount(): Int {
        return items.size
    }

    override fun onBindViewHolder(holder: ViewHolder, position: Int) {
        Glide.with(content).load(items[position].avatar).apply(RequestOptions.circleCropTransform()).into(holder.ivAvatar)

        holder.tvTitle.text = items[position].title
        holder.tvContent.text = items[position].desc
        holder.tvWatchers.text = "${items[position].watchers}"
        holder.tvCollect.text = "${items[position].collect}"
        holder.tvFork.text = "${items[position].forkNum}"
        holder.tvDate.text = items[position].date

    }


    class ViewHolder(itemView: View) : RecyclerView.ViewHolder(itemView) {
        val ivAvatar: ImageView = itemView.findViewById(R.id.iv_avatar)
        val tvTitle: TextView = itemView.findViewById(R.id.tv_title)
        val tvContent: TextView = itemView.findViewById(R.id.tv_content)
        val tvWatchers: TextView = itemView.findViewById(R.id.tv_watchers)
        val tvCollect: TextView = itemView.findViewById(R.id.tv_collect)
        val tvFork: TextView = itemView.findViewById(R.id.tv_fork)
        val tvDate: TextView = itemView.findViewById(R.id.tv_date)
    }

    interface OnItemClick{
        fun onClick(articleBean: ArticleBean, position: Int)
    }
}

有什么不懂得留言告訴我