一晤锹、前言
從現(xiàn)在開(kāi)始會(huì)偶爾寫一些付費(fèi)的內(nèi)容摩幔,之所以要寫付費(fèi)內(nèi)容,主要基于以下幾點(diǎn)鞭铆,占比權(quán)重依次降低或衡。
1、我喜歡錢车遂。
2封断、建立有用知識(shí)的門檻,把有用的東西給予真正需要的人舶担。
3坡疼、通過(guò)付費(fèi)倒逼我寫出更好的文章,如果文章不好你可以盡情吐槽衣陶,前提是你花了錢柄瑰。
二闸氮、抖音介紹
抖音現(xiàn)在真的是越來(lái)越火,我們公司的“一面APP”也想做對(duì)抖音APP的數(shù)據(jù)抓取教沾∑芽纾基于此,我下載抖音授翻,開(kāi)始研究它的HTTP請(qǐng)求內(nèi)容财骨,后來(lái)我發(fā)現(xiàn)它有毒,越刷越上癮藏姐,總感覺(jué)下一個(gè)視頻一定會(huì)更精彩隆箩,根本停不下來(lái),似乎早已忘記當(dāng)初為什么下載抖音...
當(dāng)然羔杨,最后我還是控制著了捌臊。
三、獲取抖音中某個(gè)“挑戰(zhàn)”下熱門視頻及單個(gè)視頻的URL
關(guān)于對(duì)APP進(jìn)行抓包的方法兜材,在這我不做講解理澎,這個(gè)方法在網(wǎng)上可以找到很多相關(guān)資料(收費(fèi)價(jià)值不大)。
如果我們要獲取抖音中某個(gè)“挑戰(zhàn)”下熱門視頻曙寡,大概的步驟如下:
1)配置好Fiddler或Charles(我這邊是使用Charles)糠爬,通過(guò)Charles可以對(duì)手機(jī)APP進(jìn)行抓包,如下圖1所示举庶;
2)在抖音APP上進(jìn)行挑戰(zhàn)的內(nèi)容搜索执隧,例如搜索“絕地求生”,如下圖2所示户侥;
3)點(diǎn)擊“絕地求生”挑戰(zhàn)镀琉,查看Charles中的HTTP請(qǐng)求數(shù)據(jù),找出獲取挑戰(zhàn)下視頻的API蕊唐,如下圖3所示屋摔;
我們發(fā)現(xiàn)在我們點(diǎn)擊“絕地求生”進(jìn)入這個(gè)挑戰(zhàn)的詳情頁(yè)面時(shí),抖音APP向服務(wù)端申請(qǐng)了兩個(gè)接口請(qǐng)求替梨。
一個(gè)是detail钓试,它里面包含的是這個(gè)挑戰(zhàn)的一些具體信息,因?yàn)槲倚枰氖且曨l信息副瀑,所以這個(gè)我們先不管弓熏。
另一個(gè)是aweme,它的接口內(nèi)容如下:
https://aweme.snssdk.com/aweme/v1/challenge/aweme/?iid=30373511894&device_id=35781128184&os_api=18&app_name=aweme&channel=App%20Store&idfa=811A8841-030F-4AEA-B934-C2A56489C32D&device_platform=iphone&build_number=17805&vid=A4BB3AF4-7981-4805-995B-78419881DC11&openudid=99bf2b608173e21cefd562496d2cf21fa8eba580&device_type=iPhone7,2&app_version=1.7.8&version_code=1.7.8&os_version=11.3&screen_width=750&aid=1128&ac=WIFI&ch_id=1574030716416014&count=21&cursor=0&pull_type=2&query_type=0&type=5&mas=001469ed4a7a3f61de046e21c8c7ae8bcb64a983471da18ec0c8d1&as=a1c5206d9e676a38ee4960&ts=1524500606
同時(shí)通過(guò)查看這個(gè)接口返回的JSON數(shù)據(jù)俗扇,也確定這個(gè)是獲取視頻的API硝烂,如下是返回的JSON視頻數(shù)據(jù)(只保留aweme_list中的第一條數(shù)據(jù),用于分析):
{
"has_more" : 1,
"cursor" : 21,
"rid" : "201804240023290100100511301210FF",
"status_code" : 0,
"extra" : {
"logid" : "201804240023290100100511301210FF",
"now" : 1524500609305,
"fatal_item_ids" : []
},
"aweme_list" : [
{
"text_extra" : [],
"status" : {
"is_delete" : false,
"allow_share" : true,
"is_private" : false,
"allow_comment" : true,
"private_status" : 0,
"with_goods" : false
},
"author_user_id" : 60978405851,
"is_hash_tag" : 0,
"create_time" : 1501112821,
"aweme_type" : 0,
"cha_list" : [
{
"schema" : "aweme:\/\/aweme\/challenge\/detail?cid=1574030716416014",
"user_count" : 0,
"author" : {},
"cha_name" : "絕地求生",
"type" : 0,
"desc" : "吃雞吃雞铜幽!今晚吃雞滞谢!",
"cid" : "1574030716416014"
}
],
"region" : "CN",
"user_digged" : 0,
"share_info" : {
"share_weibo_desc" : "#在抖音串稀,記錄美好生活#空氣泳",
"share_title" : "空氣泳",
"share_url" : "https:\/\/www.iesdouyin.com\/share\/video\/6447230455191178509\/?region=CN&mid=6443685673772944141&titleType=title",
"share_desc" : "在抖音,記錄美好生活"
},
"video" : {
"dynamic_cover" : {
"url_list" : [
"https:\/\/p3.pstatp.com\/obj\/30b20017abc8da2a1379",
"https:\/\/pb9.pstatp.com\/obj\/30b20017abc8da2a1379",
"https:\/\/pb3.pstatp.com\/obj\/30b20017abc8da2a1379"
],
"uri" : "30b20017abc8da2a1379"
},
"play_addr_lowbr" : {
"url_list" : [
"https:\/\/aweme.snssdk.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=0&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/api.amemv.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=0&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/aweme.snssdk.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=1&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/api.amemv.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=1&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0"
],
"uri" : "fe29c6e0c74a40f6baf22400552d6d2f"
},
"width" : 540,
"ratio" : "720p",
"play_addr" : {
"url_list" : [
"https:\/\/aweme.snssdk.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=0&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/api.amemv.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=0&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/aweme.snssdk.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=1&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/api.amemv.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=1&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0"
],
"uri" : "fe29c6e0c74a40f6baf22400552d6d2f"
},
"cover" : {
"url_list" : [
"https:\/\/p1.pstatp.com\/aweme\/300x400\/30b600031f0eca5498aa.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/300x400\/30b600031f0eca5498aa.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/300x400\/30b600031f0eca5498aa.jpeg"
],
"uri" : "300x400\/30b600031f0eca5498aa"
},
"height" : 960,
"bit_rate" : [
{
"bit_rate" : 4325045,
"gear_name" : "normal",
"quality_type" : 0
}
],
"origin_cover" : {
"url_list" : [
"http:\/\/p1.pstatp.com\/large\/30b600031f0eca5498aa.jpeg",
"http:\/\/pb3.pstatp.com\/large\/30b600031f0eca5498aa.jpeg",
"http:\/\/pb3.pstatp.com\/large\/30b600031f0eca5498aa.jpeg"
],
"uri" : "large\/30b600031f0eca5498aa"
},
"duration" : 3820,
"download_addr" : {
"url_list" : [
"https:\/\/aweme.snssdk.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=0&ratio=720p&watermark=1&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/api.amemv.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=0&ratio=720p&watermark=1&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/aweme.snssdk.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=1&ratio=720p&watermark=1&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0",
"https:\/\/api.amemv.com\/aweme\/v1\/play\/?video_id=fe29c6e0c74a40f6baf22400552d6d2f&line=1&ratio=720p&watermark=1&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0"
],
"uri" : "fe29c6e0c74a40f6baf22400552d6d2f"
},
"has_watermark" : true
},
"is_vr" : false,
"vr_type" : 0,
"share_url" : "https:\/\/www.douyin.com\/share\/video\/6447230455191178509\/?mid=6443685673772944141",
"sort_label" : "",
"music" : {
"cover_hd" : {
"url_list" : [
"http:\/\/p3.pstatp.com\/live\/1080x1080\/2bd9005342019b56cf7e.jpeg"
],
"uri" : "2bd9005342019b56cf7e"
},
"cover_large" : {
"url_list" : [
"http:\/\/p3.pstatp.com\/live\/720x720\/2bd9005342019b56cf7e.jpeg"
],
"uri" : "2bd9005342019b56cf7e"
},
"status" : 1,
"extra" : "{\"has_edited\": 0}",
"user_count" : 0,
"title" : "Iceland",
"duration" : 32,
"play_url" : {
"url_list" : [
"http:\/\/p1.pstatp.com\/obj\/29c70000654ea97a6169",
"http:\/\/p9.pstatp.com\/obj\/29c70000654ea97a6169",
"http:\/\/p3.pstatp.com\/obj\/29c70000654ea97a6169"
],
"uri" : "29c70000654ea97a6169"
},
"mid" : "6443685673772944141",
"offline_desc" : "",
"schema_url" : "",
"cover_medium" : {
"url_list" : [
"http:\/\/p3.pstatp.com\/live\/200x200\/2bd9005342019b56cf7e.jpeg"
],
"uri" : "2bd9005342019b56cf7e"
},
"is_original" : false,
"id" : 6443685673772944141,
"cover_thumb" : {
"url_list" : [
"http:\/\/p3.pstatp.com\/live\/100x100\/2bd9005342019b56cf7e.jpeg"
],
"uri" : "2bd9005342019b56cf7e"
},
"audio_track" : {
"url_list" : [
"https:\/\/p1.pstatp.com\/obj\/46bd000dbd65c3dfd8ba",
"https:\/\/pb3.pstatp.com\/obj\/46bd000dbd65c3dfd8ba",
"https:\/\/pb3.pstatp.com\/obj\/46bd000dbd65c3dfd8ba"
],
"uri" : "46bd000dbd65c3dfd8ba"
},
"effects_data" : {
"url_list" : [
"https:\/\/p1.pstatp.com\/obj\/2efe00235ebf732bb56f",
"https:\/\/pb3.pstatp.com\/obj\/2efe00235ebf732bb56f",
"https:\/\/pb3.pstatp.com\/obj\/2efe00235ebf732bb56f"
],
"uri" : "2efe00235ebf732bb56f"
},
"source_platform" : 22,
"author" : "FlyBoy,Gavrielle",
"id_str" : "6443685673772944141"
},
"is_top" : 0,
"aweme_id" : "6447230455191178509",
"cmt_swt" : false,
"risk_infos" : {
"warn" : false,
"content" : "",
"risk_sink" : false,
"type" : 0
},
"label_origin_author" : {
"url_list" : [
"https:\/\/p3.pstatp.com\/obj\/216a001ff805affc1dab",
"https:\/\/pb9.pstatp.com\/obj\/216a001ff805affc1dab",
"https:\/\/pb3.pstatp.com\/obj\/216a001ff805affc1dab"
],
"uri" : "216a001ff805affc1dab"
},
"label_origin_author_text" : "發(fā)起者",
"label_top" : {
"url_list" : [
"https:\/\/p3.pstatp.com\/obj\/c150000f34767e2cb56",
"https:\/\/pb9.pstatp.com\/obj\/c150000f34767e2cb56",
"https:\/\/pb3.pstatp.com\/obj\/c150000f34767e2cb56"
],
"uri" : "c150000f34767e2cb56"
},
"is_ads" : false,
"statistics" : {
"digg_count" : 19578,
"aweme_id" : "6447230455191178509",
"share_count" : 283,
"play_count" : 1296122,
"comment_count" : 100
},
"video_labels" : [],
"rate" : 12,
"bodydance_score" : 0,
"desc" : "空氣泳",
"author" : {
"special_lock" : 0,
"is_binded_weibo" : false,
"shield_follow_notice" : 0,
"nickname_lock" : 0,
"avatar_larger" : {
"url_list" : [
"https:\/\/p1.pstatp.com\/aweme\/1080x1080\/38c90028f622dd195aaf.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/1080x1080\/38c90028f622dd195aaf.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/1080x1080\/38c90028f622dd195aaf.jpeg"
],
"uri" : "38c90028f622dd195aaf"
},
"follow_status" : 0,
"with_commerce_entry" : false,
"original_music_qrcode" : null,
"authority_status" : 0,
"is_ad_fake" : false,
"prevent_download" : false,
"enterprise_verify" : false,
"verification_type" : 1,
"app_id" : 1128,
"weibo_url" : "",
"community_discipline_status" : 0,
"risk_flag" : 0,
"need_recommend" : 1,
"live_rec_level" : 0,
"unique_id" : "MysticzZ",
"hide_location" : false,
"enterprise_verify_reason" : "",
"short_id" : "17612748",
"account_region" : "",
"star_use_new_download" : false,
"signature" : "頹",
"avatar_medium" : {
"url_list" : [
"https:\/\/p1.pstatp.com\/aweme\/720x720\/38c90028f622dd195aaf.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/720x720\/38c90028f622dd195aaf.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/720x720\/38c90028f622dd195aaf.jpeg"
],
"uri" : "38c90028f622dd195aaf"
},
"verify_info" : "",
"create_time" : 1495648525,
"story_open" : false,
"region" : "CN",
"hide_search" : false,
"avatar_thumb" : {
"url_list" : [
"https:\/\/p1.pstatp.com\/aweme\/100x100\/38c90028f622dd195aaf.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/100x100\/38c90028f622dd195aaf.jpeg",
"https:\/\/pb3.pstatp.com\/aweme\/100x100\/38c90028f622dd195aaf.jpeg"
],
"uri" : "38c90028f622dd195aaf"
},
"school_poi_id" : "",
"update_before" : 0,
"shield_comment_notice" : 0,
"original_music_cover" : null,
"shield_digg_notice" : 0,
"bind_phone" : "",
"live_verify" : 0,
"birthday" : "1999-03-28",
"school_type" : 0,
"uid" : "60978405851",
"avatar_uri" : "38c90028f622dd195aaf",
"weibo_schema" : "",
"is_verified" : true,
"custom_verify" : "",
"commerce_user_level" : 0,
"gender" : 1,
"has_register_notice" : 0,
"reflow_page_gid" : 0,
"reflow_page_uid" : 0,
"nickname" : "Atlantis.",
"weibo_verify" : "",
"language" : "zh-Hans",
"share_qrcode_uri" : "216a0035fc39c9e9376f",
"room_id" : 0,
"constellation" : 1,
"school_name" : "",
"weibo_name" : ""
},
"is_relieve" : false
},
{
...第二條數(shù)據(jù)
}
...其他更多的數(shù)據(jù)
]
}
4)對(duì)這個(gè)API進(jìn)行分析(破解或者繞過(guò)API的加密)狮杨,找出可以直接獲取JSON數(shù)據(jù)的純種API母截;
API的內(nèi)容為:
https://aweme.snssdk.com/aweme/v1/challenge/aweme/?iid=30373511894&device_id=35781128184&os_api=18&app_name=aweme&channel=App%20Store&idfa=811A8841-030F-4AEA-B934-C2A56489C32D&device_platform=iphone&build_number=17805&vid=A4BB3AF4-7981-4805-995B-78419881DC11&openudid=99bf2b608173e21cefd562496d2cf21fa8eba580&device_type=iPhone7,2&app_version=1.7.8&version_code=1.7.8&os_version=11.3&screen_width=750&aid=1128&ac=WIFI&ch_id=1574030716416014&count=21&cursor=0&pull_type=2&query_type=0&type=5&mas=001469ed4a7a3f61de046e21c8c7ae8bcb64a983471da18ec0c8d1&as=a1c5206d9e676a38ee4960&ts=1524500606
。
為了便于分析橄教,我們先把一些無(wú)關(guān)的手機(jī)設(shè)備和版本信息除掉: https://aweme.snssdk.com/aweme/v1/challenge/aweme/?ch_id=1574030716416014&count=21&cursor=0&pull_type=2&query_type=0&type=5&as=a1c5206d9e676a38ee4960&ts=1524500606
它現(xiàn)在只剩下ch_id清寇、count、cursor护蝶、pull_type华烟、query_type、type持灰、as以及ts幾個(gè)參數(shù)盔夜,其中pull_type、query_type和type三個(gè)是固定的堤魁,不用管喂链。cursor和count是獲取的數(shù)據(jù)的開(kāi)始和數(shù)量,ch_id應(yīng)該是挑戰(zhàn)的唯一標(biāo)示妥泉。接下來(lái)就是as和ts兩個(gè)參數(shù)椭微,通過(guò)多次實(shí)驗(yàn),我們發(fā)現(xiàn)這個(gè)API使用了as和cp參數(shù)進(jìn)行了加密處理盲链,也就是說(shuō)我們無(wú)法使用這個(gè)API直接獲取到API的數(shù)據(jù)蝇率,需要對(duì)as、ts的加密算法進(jìn)行破解匈仗。當(dāng)然瓢剿,上面我也提過(guò),面對(duì)這個(gè)情況悠轩,要么破解算法要么繞過(guò)加密,原則上是能夠繞過(guò)的就不要選擇破解攻泼,因?yàn)槠平獾碾y度很大火架,需要逆向,這個(gè)以后再講(我也沒(méi)做過(guò))忙菠。
APP或PC上破解不了的接口何鸡,可以嘗試使用m站。這個(gè)是我們的一貫原則牛欢。
我們發(fā)現(xiàn)抖音中的挑戰(zhàn)有一個(gè)分享按鈕骡男,我們?cè)囍窒沓鲞@個(gè)H5頁(yè)面來(lái)進(jìn)行分析,分享之后的URL為:
https://www.iesdouyin.com/share/challenge/1574030716416014?utm_campaign=client_share&app=aweme&utm_medium=ios&iid=30373511894&utm_source=weixin&tt_from=weixin&utm_source=weixin&utm_medium=aweme_ios&utm_campaign=client_share&uid=58875748483&did=30373511894
在瀏覽器上打開(kāi)這個(gè)URL傍睹,通過(guò)分析它的請(qǐng)求隔盛,只看XHR犹菱,見(jiàn)下圖4: