Spark 應(yīng)用監(jiān)控告警和自動(dòng)重啟

Spark on yarn 執(zhí)行流計(jì)算時(shí)持钉，如果流掛了帅戒，沒有提醒會(huì)導(dǎo)致實(shí)時(shí)指標(biāo)計(jì)算停滯膘魄，為了保證流的7/24運(yùn)行昙读，需要有一個(gè)能監(jiān)控Spark on yarn上的應(yīng)用召调，實(shí)現(xiàn)失敗重啟、失敗告警蛮浑，同時(shí)能展示Spark應(yīng)用的相關(guān)指標(biāo)唠叛，實(shí)現(xiàn)數(shù)據(jù)質(zhì)量的監(jiān)控和管理。

一沮稚、指標(biāo)監(jiān)控模塊

使用Prometheus玻墅、graphite_exporter、Grafana實(shí)現(xiàn)Spark應(yīng)用指標(biāo)監(jiān)控壮虫。

1.Spark配置Graphite metrics

spark 是自帶 Graphite Sink 的，只需要配置一下metrics.properties:

*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink  
*.sink.graphite.protocol=tcp 
*.sink.graphite.host=127.0.0.1 
*.sink.graphite.port=9109 
*.sink.graphite.period=5 
*.sink.graphite.unit=seconds 
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource 
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

提交時(shí)記得使用 --files /path/to/spark/conf/metrics.properties 參數(shù)將配置文件分發(fā)到所有的 Executor。

2.安裝啟動(dòng)graphite_exporter

prometheus 提供了一個(gè)插件（graphite_exporter）囚似，可以將 Graphite metrics 進(jìn)行轉(zhuǎn)化并寫入 Prometheus (本文的方式)剩拢，

先去https://prometheus.io/download/下載graphite_exporter

image.png

啟動(dòng) graphite_exporter 時(shí)加載配置文件

./graphite_exporter --graphite.mapping-config=graphite_exporter_mapping

graphite_exporter_mapping：

mappings:
- match: '*.*.executor.filesystem.*.*'
  name: filesystem_usage
  labels:
    application: $1
    executor_id: $2
    fs_type: $3
    qty: $4

- match: '*.*.jvm.*.*'
  name: jvm_memory_usage
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4

- match: '*.*.executor.jvmGCTime.count'
  name: jvm_gcTime_count
  labels:
    application: $1
    executor_id: $2

- match: '*.*.jvm.pools.*.*'
  name: jvm_memory_pools
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4

- match: '*.*.executor.threadpool.*'
  name: executor_tasks
  labels:
    application: $1
    executor_id: $2
    qty: $3

- match: '*.*.BlockManager.*.*'
  name: block_manager
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4

- match: DAGScheduler.*.*
  name: DAG_scheduler
  labels:
    type: $1
    qty: $2

安裝啟動(dòng)應(yīng)用后，如果采集成功饶唤，將在 http://127.0.0.1:9108/metrics 頁面中看到相應(yīng)的信息徐伐。

3.配置 Prometheus和Grafana

修改/path/to/prometheus/prometheus.yml，增加

scrape_configs: - job_name: 'spark' static_configs: - targets: ['localhost:9108']

配置Grafana并加載Json配置:

spark_prometheus.json

{ "id": 1, "title": "Spark Prometheus", "originalTitle": "Spark Prometheus", "tags": [], "style": "dark", "timezone": "browser", "editable": true, "hideControls": false, "sharedCrosshair": false, "rows": [ { "collapse": false, "editable": true, "height": "250px", "panels": [ { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 0, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": 0, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 1, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "alias": "total", "color": "#F2C96D", "linewidth": 7, "yaxis": 2, "zindex": 3 } ], "span": 4, "stack": false, "steppedLine": false, "targets": [ { "expr": "filesystem_usage{exported_job=\"read_bytes\", fs_type=\"hdfs\", application=\"$application_ID\"}", "intervalFactor": 2, "legendFormat": "{{executor_id}}", "metric": "filesystem_usage", "refId": "A", "step": 2, "target": "" }, { "expr": "sum(filesystem_usage{exported_job=\"read_bytes\", fs_type=\"hdfs\", application=\"$application_ID\"})", "intervalFactor": 2, "legendFormat": "total", "metric": "filesystem_usage", "refId": "B", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Executor HDFS reads", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] }, { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 1, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": null, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 2, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": false, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 4, "stack": true, "steppedLine": false, "targets": [ { "expr": "filesystem_usage{exported_job=\"write_bytes\", fs_type=\"hdfs\", application=\"$application_ID\"}", "intervalFactor": 2, "legendFormat": "{{executor_id}}", "metric": "filesystem_usage", "refId": "A", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Executor HDFS writes", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] }, { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 1, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": null, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 3, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "alias": "total", "yaxis": 2 } ], "span": 4, "stack": true, "steppedLine": false, "targets": [ { "expr": "avg(rate(filesystem_usage{exported_job=\"read_bytes\", fs_type=\"hdfs\", application=\"$application_ID\"}[1m]))", "intervalFactor": 1, "legendFormat": "average", "metric": "filesystem_usage", "refId": "A", "step": 1, "target": "" }, { "expr": "sum(rate(filesystem_usage{exported_job=\"read_bytes\", fs_type=\"hdfs\", application=\"$application_ID\"}[1m]))", "intervalFactor": 1, "legendFormat": "total", "metric": "filesystem_usage", "refId": "B", "step": 1, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "HDFS Read Rate / s", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] } ], "title": "HDFS stats" }, { "collapse": false, "editable": true, "height": "250px", "panels": [ { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 0, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": 0, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 4, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 6, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_usage{mem_type=\"heap\", qty=\"usage\", application=\"$application_ID\", executor_id=\"driver\"}", "intervalFactor": 2, "legendFormat": "{{executor_id}}", "metric": "jvm_memory_usage", "refId": "A", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Driver Heap Usage", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] }, { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 1, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": null, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 5, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 6, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_pools{executor_id=\"driver\", application=\"$application_ID\",qty=\"used\"}", "intervalFactor": 2, "legendFormat": "{{mem_type}}", "metric": "jvm_memory_pools", "refId": "A", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Driver JVM Memory Pools", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] }, { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 0, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": null, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 6, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": false, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 4, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_usage{mem_type=\"heap\", qty=\"usage\", application=\"$application_ID\", executor_id!~\"driver\"} ", "intervalFactor": 2, "legendFormat": "{{executor_id}}", "metric": "jvm_memory_usage", "refId": "A", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Executor Heap Usage", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] }, { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 0, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": null, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 7, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": false, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 4, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_pools{executor_id!~\"driver\", application=\"$application_ID\",qty=\"usage\", mem_type=\"PS-Eden-Space\"}", "intervalFactor": 2, "legendFormat": "{{executor_id}}", "metric": "jvm_memory_pools", "refId": "A", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Executor Eden-Space", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] }, { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 0, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": null, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 9, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 4, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_pools{executor_id!~\"driver\", application=\"$application_ID\",qty=\"usage\", mem_type=\"PS-Old-Gen\"}", "intervalFactor": 2, "legendFormat": "{{executor_id}}", "metric": "jvm_memory_pools", "refId": "A", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Executor Old-Gen", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] }, { "aliasColors": {}, "bars": false, "datasource": "Prometheus", "editable": true, "error": false, "fill": 1, "grid": { "leftLogBase": 1, "leftMax": null, "leftMin": null, "rightLogBase": 1, "rightMax": null, "rightMin": null, "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 10, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "alias": "completeTasks", "yaxis": 2 } ], "span": 12, "stack": false, "steppedLine": false, "targets": [ { "expr": "sum(executor_tasks{application=\"$application_ID\", qty!~\"maxPool_size\"}) by (qty)", "intervalFactor": 2, "legendFormat": "{{qty}}", "refId": "B", "step": 2, "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Tasks", "tooltip": { "shared": true, "value_type": "cumulative" }, "type": "graph", "x-axis": true, "y-axis": true, "y_formats": [ "short", "short" ] } ], "title": "New row" } ], "time": { "from": "now-5m", "to": "now" }, "timepicker": { "now": true, "refresh_intervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "time_options": [ "5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d" ] }, "templating": { "list": [ { "allFormat": "glob", "current": { "text": "application_1564363240533_7587", "value": "application_1564363240533_7587" }, "datasource": null, "includeAll": false, "multi": false, "multiFormat": "glob", "name": "application_ID", "options": [ { "text": "application_1564363240533_7587", "value": "application_1564363240533_7587", "selected": false }, { "text": "application_1564363240533_7592", "value": "application_1564363240533_7592", "selected": false }, { "text": "application_1564363240533_7615", "value": "application_1564363240533_7615", "selected": false } ], "query": "label_values(application)", "refresh": true, "refresh_on_load": false, "type": "query" } ] }, "annotations": { "list": [] }, "refresh": "5s", "schemaVersion": 8, "version": 0, "links": [] }

二募狂、進(jìn)程監(jiān)控失敗重啟和告警模塊

監(jiān)控yarn上指定的Spark應(yīng)用是否存在办素，不存在則發(fā)出告警。

使用Python腳本查看yarn狀態(tài)祸穷，指定監(jiān)控應(yīng)用性穿，應(yīng)用中斷則通過webhook發(fā)送報(bào)警信息到釘釘群，并且自動(dòng)重啟雷滚。

#!/usr/bin/python3.5
# -*- coding: utf-8 -*-
import os
import json
import requests

'''
Yarn應(yīng)用監(jiān)控：當(dāng)配置的應(yīng)用名不在yarn applicaition -list時(shí)需曾，釘釘告警
'''


def yarn_list(applicatin_list):
    yarn_application_list = os.popen('yarn application -list').read()
    result = ""
    for appName in applicatin_list:
        if appName in yarn_application_list:
            print("應(yīng)用:%s 正常!" % appName)
        else:
            result += ("告警--應(yīng)用:%s 中斷!" % appName)
            if "應(yīng)用名1" == appName:
                os.system('重啟命令')

    return result


def dingding_robot(data):
    # 機(jī)器人的webhooK 獲取地址參考：https://open-doc.dingtalk.com/microapp/serverapi2/qf2nxq
    webhook = "https://oapi.dingtalk.com/robot/send?access_token" \
              "=你的token "
    headers = {'content-type': 'application/json'}  # 請(qǐng)求頭
    r = requests.post(webhook, headers=headers, data=json.dumps(data))
    r.encoding = 'utf-8'
    return r.text


if __name__ == '__main__':
    applicatin_list = ["應(yīng)用名1", "應(yīng)用名2", "應(yīng)用名3"]
    output = yarn_list(applicatin_list)
    print(output)

    if len(output) > 0:
        # 請(qǐng)求參數(shù) 可以寫入配置文件中
        data = {
            "msgtype": "text",
            "text": {
                "content": output
            },
            "at": {
                "atMobiles": [
                    "xxxxxxx"
                ],
                "isAtAll": False
            }
        }
        res = dingding_robot(data)
        print(res)  # 打印請(qǐng)求結(jié)果
    else:
        print("一切正常!")

三、指標(biāo)監(jiān)控告警

TBC:使用Prometheus的AlertManager模塊實(shí)現(xiàn)指標(biāo)的監(jiān)控祈远。

最后編輯于：2019.08.22 15:24:13

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者

人面猴
序言：七十年代末呆万，一起剝皮案震驚了整個(gè)濱河市，隨后出現(xiàn)的幾起案子车份，更是在濱河造成了極大的恐慌谋减，老刑警劉巖，帶你破解...
沈念sama閱讀 212,383評(píng)論 6贊 493
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件扫沼，死亡現(xiàn)場離奇詭異出爹，居然都是意外死亡，警方通過查閱死者的電腦和手機(jī)充甚，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 90,522評(píng)論 3贊 385
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門以政，熙熙樓的掌柜王于貴愁眉苦臉地迎上來，“玉大人伴找，你說我怎么就攤上這事盈蛮。” “怎么了技矮？”我有些...
開封第一講書人閱讀 157,852評(píng)論 0贊 348
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵抖誉，是天一觀的道長。經(jīng)常有香客問我衰倦，道長袒炉，這世上最難降的妖魔是什么？我笑而不...
開封第一講書人閱讀 56,621評(píng)論 1贊 284
?港島之戀（遺憾婚禮）
正文為了忘掉前任樊零，我火速辦了婚禮我磁，結(jié)果婚禮上孽文，老公的妹妹穿的比我還像新娘。我一直安慰自己夺艰，他們只是感情好芋哭，可當(dāng)我...
茶點(diǎn)故事閱讀 65,741評(píng)論 6贊 386
惡毒庶女頂嫁案：這布局不是一般人想出來的
文/花漫我一把揭開白布。她就那樣靜靜地躺著郁副，像睡著了一般减牺。火紅的嫁衣襯著肌膚如雪。梳的紋絲不亂的頭發(fā)上存谎，一...
開封第一講書人閱讀 49,929評(píng)論 1贊 290
城市分裂傳說
那天拔疚，我揣著相機(jī)與錄音，去河邊找鬼既荚。笑死稚失，一個(gè)胖子當(dāng)著我的面吹牛，可吹牛的內(nèi)容都是我干的固以。我是一名探鬼主播墩虹，決...
沈念sama閱讀 39,076評(píng)論 3贊 410
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼，長吁一口氣：“原來是場噩夢啊……” “哼憨琳！你這毒婦竟也來了诫钓？” 一聲冷哼從身側(cè)響起，我...
開封第一講書人閱讀 37,803評(píng)論 0贊 268
萬榮殺人案實(shí)錄
序言：老撾萬榮一對(duì)情侶失蹤篙螟，失蹤者是張志新（化名）和其女友劉穎菌湃，沒想到半個(gè)月后，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體遍略，經(jīng)...
沈念sama閱讀 44,265評(píng)論 1贊 303
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡惧所，尸身上長有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 36,582評(píng)論 2贊 327
?白月光啟示錄
正文我和宋清朗相戀三年，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了绪杏。大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片下愈。...
茶點(diǎn)故事閱讀 38,716評(píng)論 1贊 341
活死人
序言：一個(gè)原本活蹦亂跳的男人離奇死亡，死狀恐怖蕾久，靈堂內(nèi)的尸體忽然破棺而出势似，到底是詐尸還是另有隱情，我是刑警寧澤僧著，帶...
沈念sama閱讀 34,395評(píng)論 4贊 333
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布履因，位于F島的核電站，受9級(jí)特大地震影響盹愚，放射性物質(zhì)發(fā)生泄漏栅迄。R本人自食惡果不足惜，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 40,039評(píng)論 3贊 316
男人毒藥：我在死后第九天來索命
文/蒙蒙一皆怕、第九天我趴在偏房一處隱蔽的房頂上張望毅舆。院中可真熱鬧西篓，春花似錦、人聲如沸朗兵。這莊子的主人今日做“春日...
開封第一講書人閱讀 30,798評(píng)論 0贊 21
一樁弒父案，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽余掖。三九已至，卻和暖如春礁鲁，著一層夾襖步出監(jiān)牢的瞬間盐欺，已是汗流浹背。一陣腳步聲響...
開封第一講書人閱讀 32,027評(píng)論 1贊 266
情欲美人皮
我被黑心中介騙來泰國打工仅醇，沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留冗美，地道東北人。一個(gè)月前我還...
沈念sama閱讀 46,488評(píng)論 2贊 361
代替公主和親
正文我出身青樓析二，卻偏偏與公主長得像粉洼，于是被迫代替她去往敵國和親。傳聞我的和親對(duì)象是個(gè)殘疾皇子叶摄，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 43,612評(píng)論 2贊 350