Ubuntu系統(tǒng)使用SPDK運(yùn)行fio測(cè)試

??因最近需求所要褥实，所以研究了一下SPDK運(yùn)行Fio方式，本文主要講解spdk_nvme的方式運(yùn)行fio進(jìn)行性能測(cè)試赃春。若感興趣可以看看官方文檔【SPDK：程序員指南】

一、拉取SPDK，以及準(zhǔn)備環(huán)境

????下載SPDK軟件包鼎姊，--recursive參數(shù)是會(huì)循環(huán)拉去SPDK，相當(dāng)于“git clone https://github.com/spdk/spdk”之后，在進(jìn)行“git submodule update --init”相寇。
????下載Fio慰于，需注意Fio的版本需要同SPDK里面一致乍桂，可查看在SPDK路徑“./examples/nvme/fio_plugin/”里面的“README.md”文件俄精，按文件中的指令進(jìn)行g(shù)it拉取fio代碼。

root@test-System-Product-Name:/home# git clone https://github.com/spdk/spdk --recursive
Cloning into 'spdk'...
remote: Enumerating objects: 181402, done.
remote: Counting objects: 100% (5702/5702), done.
remote: Compressing objects: 100% (2099/2099), done.
remote: Total 181402 (delta 3945), reused 4698 (delta 3569), pack-reused 175700
Receiving objects: 100% (181402/181402), 79.23 MiB | 29.00 KiB/s, done.
Resolving deltas: 100% (141210/141210), done.
Submodule 'dpdk' (https://github.com/spdk/dpdk.git) registered for path 'dpdk'
Submodule 'intel-ipsec-mb' (https://github.com/spdk/intel-ipsec-mb.git) registered for path 'intel-ipsec-mb'
Submodule 'isa-l' (https://github.com/spdk/isa-l.git) registered for path 'isa-l'
Submodule 'isa-l-crypto' (https://github.com/intel/isa-l_crypto) registered for path 'isa-l-crypto'
Submodule 'libvfio-user' (https://github.com/nutanix/libvfio-user.git) registered for path 'libvfio-user'
Submodule 'ocf' (https://github.com/Open-CAS/ocf.git) registered for path 'ocf'
Submodule 'xnvme' (https://github.com/OpenMPDK/xNVMe.git) registered for path 'xnvme'
Cloning into '/root/spdk/dpdk'...
......
root@test-System-Product-Name:/home# git clone https://github.com/axboe/fio
Cloning into 'fio'...
remote: Enumerating objects: 35674, done.
remote: Counting objects: 100% (35674/35674), done.
remote: Compressing objects: 100% (10580/10580), done.
remote: Total 35674 (delta 25041), reused 35351 (delta 24999), pack-reused 0
Receiving objects: 100% (35674/35674), 25.53 MiB | 2.67 MiB/s, done.
Resolving deltas: 100% (25041/25041), done.
root@test-System-Product-Name:/home# cd fio/

# git拉去的回事mester分支fio海洼，若拉取固定版本需chekout
root@test-System-Product-Name:/home# git checkout fio-3.17
root@test-System-Product-Name:/home/fio# ./configure
Operating system              Linux
CPU                           x86_64
......

TCMalloc support              no
seed_buckets                  4
root@test-System-Product-Name:/home/fio#
root@test-System-Product-Name:/home/fio# make
FIO_VERSION = fio-3.35-30-gedaee
    CC crc/crc16.o
......
  LINK unittests/unittest
root@test-System-Product-Name:/home/fio#
二佳励、編譯SPDK

????編譯SPDK完成后休里，可以查看SPDK下“/build/fio/”目錄，里面是否生成spdk_bdev和spdk_nvme兩個(gè)二進(jìn)制文件赃承。spdk_nvme是基于裸盤(pán)NVMe的fio_plugin妙黍，其特點(diǎn)為I/O通過(guò)SPDK用戶態(tài)驅(qū)動(dòng)直接訪問(wèn)裸盤(pán)，常用于評(píng)估SPDK用戶態(tài)驅(qū)動(dòng)在裸盤(pán)上的性能瞧剖。spdk_bdev是基于bdev的fio_plugin拭嫁，其特點(diǎn)為I/O測(cè)試基于SPDK塊設(shè)備bdev之上，所有I/O經(jīng)由塊設(shè)備層bdev筒繁，再傳送至裸盤(pán)設(shè)備噩凹。常用于評(píng)估SPDK塊設(shè)備bdev的性能。

root@test-System-Product-Name:/home# cd spdk/
# 運(yùn)行scripts目錄下的pkgdep.sh文件獲取SPDK所需的依賴(lài)包
root@test-System-Product-Name:/home/spdk# ./scripts/pkgdep.sh
root@test-System-Product-Name:/home/spdk# ls
app              autorun.sh    CODE_OF_CONDUCT.md  deprecation.md  dpdkbuild  include         isalbuild        libvfio-user  mdl_rules.rb  proto      scripts      vfiouserbuild
autobuild.sh     autotest.sh   CONFIG              doc             examples   intel-ipsec-mb  isa-l-crypto     LICENSE       mk            python     SECURITY.md  xnvme
autopackage.sh   build         configure           docker          fio.fio    ipsecbuild      isalcryptobuild  licenses      module        README.md  shared_lib   xnvmebuild
autorun_post.py  CHANGELOG.md  CONTRIBUTING.md     dpdk            go         isa-l           lib              Makefile      ocf           rpmbuild   test
root@test-System-Product-Name:/home/spdk#
# 這里fio需要選擇剛拉取并make編譯的fio路徑毡咏，等待完成
root@test-System-Product-Name:/home/spdk# ./configure --with-fio=/home/fio
root@test-System-Product-Name:/home/spdk# make
root@test-System-Product-Name:/home/spdk# ls -l /home/spdk/build/fio/
total 18556
-rwxr-xr-x 1 root root 13215920 6月   8 15:22 spdk_bdev
-rwxr-xr-x 1 root root  5783136 6月   8 15:22 spdk_nvme
root@test-System-Product-Name:/home/spdk#
三驮宴、運(yùn)行SPDK和fio

??3.1、SDPK安裝完成后呕缭，需查看一下是否可以正常運(yùn)行堵泽。可以運(yùn)行SPDK目錄下的“scripts/setup.sh”文件恢总，當(dāng)前驅(qū)動(dòng)是否替換為SPDK的驅(qū)動(dòng)迎罗。

root@test-System-Product-Name:/home/spdk# scripts/setup.sh
0000:03:00.0 (1dee 5216): Already using the uio_pci_generic driver
INFO: Requested 1024 hugepages but 5738 already allocated on node0
# status：可以查看SPDK驅(qū)動(dòng)下盤(pán)的BDF號(hào)，以及狀態(tài)
root@test-System-Product-Name:/home/spdk# scripts/setup.sh status
Hugepages
node     hugesize     free /  total
node0   1048576kB        1 /      1
node0      2048kB     5738 /   5738

Type     BDF             Vendor Device NUMA    Driver           Device     Block devices
NVMe     0000:03:00.0    1dee   5216   0       uio_pci_generic  -          -
# reset：恢復(fù)nvme驅(qū)動(dòng)
root@test-System-Product-Name:/home/spdk# scripts/setup.sh reset
0000:03:00.0 (1dee 5216): uio_pci_generic -> nvme
root@test-System-Product-Name:/home/spdk#

??3.2片仿、運(yùn)行fio文件纹安，當(dāng)前介紹的fio運(yùn)行方式主要是以spdk_nvme作為驅(qū)動(dòng)方式。NVMe的fio_plugin支持兩種模式下的測(cè)試砂豌，一是本地的NVMe設(shè)備厢岂，即NVMe over PCIe；二是遠(yuǎn)端的NVMe設(shè)備阳距，即NVMe over Fabrics塔粒。二者運(yùn)行方式區(qū)別主要在于filename參數(shù)配置不同。下面是fio文件格式筐摘，和我們平常寫(xiě)fio的方式一樣卒茬。
??需要注意：使用spdk_nvme方式需setup替換驅(qū)動(dòng)船老；必須設(shè)置“thread=1”參數(shù)；以及盤(pán)的BDF號(hào)圃酵，也就是filename參數(shù)按這個(gè)格式進(jìn)行編寫(xiě)“filename=trtype=PCIe traddr=0000.03.00.0 ns=1”柳畔。

[global]
ioengine=spdk
thread=1
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=2
iodepth=128
size=4
numjobs=1
[job1]
rw=randread
filename=trtype=PCIe traddr=0000.03.00.0 ns=1
filename=trtype=TCP adrfam=IPv4 traddr=192.168.100.100 trsvcid=4420 ns=1

??3.3、終端演示運(yùn)行
????方式一辜昵、fio文件只需寫(xiě)關(guān)鍵參數(shù)荸镊，filename放在外面運(yùn)行，需注意此時(shí)filename需加引號(hào)“ '--filename=trtype=PCIe traddr=0000.03.00.0 ns=1' ”堪置。

root@test-System-Product-Name:/home# LD_PRELOAD=/home/spdk/build/fio/spdk_nvme /home/fio/fio /home/spdk/examples/nvme/fio_plugin/example_config.fio '--filename=trtype=PCIe traddr=0000.03.00.0 ns=1'
job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][75.0%][r=325MiB/s][r=83.3k IOPS][eta 00m:01s]
job1: (groupid=0, jobs=1): err= 0: pid=423667: Fri Jun  9 09:30:38 2023
  read: IOPS=85.7k, BW=335MiB/s (351MB/s)(670MiB/2002msec)
    slat (nsec): min=75, max=1513, avg=80.92, stdev= 6.39
    clat (usec): min=95, max=65450, avg=1483.68, stdev=1437.30
     lat (usec): min=95, max=65450, avg=1483.76, stdev=1437.30
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  775], 10.00th=[  881], 20.00th=[ 1045],
     | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1418], 60.00th=[ 1532],
     | 70.00th=[ 1647], 80.00th=[ 1811], 90.00th=[ 2008], 95.00th=[ 2180],
     | 99.00th=[ 2638], 99.50th=[ 2900], 99.90th=[18744], 99.95th=[46924],
     | 99.99th=[55313]
   bw (  KiB/s): min=312808, max=356632, per=100.00%, avg=342912.00, stdev=20345.47, samples=4
   iops        : min=78202, max=89158, avg=85728.00, stdev=5086.37, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=3.99%, 1000=13.23%
  lat (msec)   : 2=72.35%, 4=10.26%, 10=0.01%, 20=0.07%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu          : usr=99.80%, sys=0.15%, ctx=3, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=171583,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), io=670MiB (703MB), run=2002-2002msec

????方式二、filename也放在fio文件里面张惹，格式如上3.2所示舀锨。

root@test-System-Product-Name:/home# LD_PRELOAD=/home/spdk/build/fio/spdk_nvme /home/fio/fio /home/spdk/examples/nvme/fio_plugin/example_config.fio
job1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][75.0%][r=347MiB/s][r=88.9k IOPS][eta 00m:01s]
job1: (groupid=0, jobs=1): err= 0: pid=421306: Thu Jun  8 18:03:19 2023
  read: IOPS=88.5k, BW=346MiB/s (363MB/s)(692MiB/2002msec)
    slat (nsec): min=75, max=1503, avg=81.59, stdev= 9.00
    clat (usec): min=98, max=4054, avg=1436.34, stdev=440.02
     lat (usec): min=98, max=4054, avg=1436.43, stdev=440.02
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  775], 10.00th=[  881], 20.00th=[ 1045],
     | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1418], 60.00th=[ 1532],
     | 70.00th=[ 1647], 80.00th=[ 1795], 90.00th=[ 2008], 95.00th=[ 2180],
     | 99.00th=[ 2606], 99.50th=[ 2802], 99.90th=[ 3261], 99.95th=[ 3458],
     | 99.99th=[ 3818]
   bw (  KiB/s): min=348552, max=357032, per=100.00%, avg=354222.00, stdev=3911.61, samples=4
   iops        : min=87138, max=89258, avg=88555.50, stdev=977.90, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=4.01%, 1000=13.21%
  lat (msec)   : 2=72.45%, 4=10.30%, 10=0.01%
  cpu          : usr=100.00%, sys=0.00%, ctx=1, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=177238,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=346MiB/s (363MB/s), 346MiB/s-346MiB/s (363MB/s-363MB/s), io=692MiB (726MB), run=2002-2002msec

????方式三、寫(xiě)成我們平常使用的fio形式宛逗，即命令行形式運(yùn)行坎匿。需注意當(dāng)前filename后面參數(shù)需要加引號(hào)“ --filename='trtype=PCIe traddr=0000.03.00.0 ns=1' ”。

root@test-System-Product-Name:/home# LD_PRELOAD=/home/spdk/build/fio/spdk_nvme /home/fio/fio --ioengine=spdk --bs=4k --rw=randread --thread=1 --group_reporting=1 --direct=1 --verify=0 --time_based=1 --ramp_time=0 --runtime=2 --iodepth=128 --numjobs=1 --size=1G --name=job --filename='trtype=PCIe traddr=0000.03.00.0 ns=1'
job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][-.-%][r=348MiB/s][r=89.0k IOPS][eta 00m:00s]
job: (groupid=0, jobs=1): err= 0: pid=423702: Fri Jun  9 09:41:45 2023
  read: IOPS=85.7k, BW=335MiB/s (351MB/s)(670MiB/2002msec)
    slat (nsec): min=74, max=1673, avg=86.59, stdev= 8.44
    clat (usec): min=93, max=66157, avg=1483.63, stdev=1428.75
     lat (usec): min=94, max=66157, avg=1483.72, stdev=1428.75
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  775], 10.00th=[  881], 20.00th=[ 1037],
     | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1418], 60.00th=[ 1532],
     | 70.00th=[ 1663], 80.00th=[ 1811], 90.00th=[ 2024], 95.00th=[ 2212],
     | 99.00th=[ 2638], 99.50th=[ 2900], 99.90th=[18744], 99.95th=[46924],
     | 99.99th=[54789]
   bw (  KiB/s): min=311424, max=358440, per=100.00%, avg=342920.00, stdev=21436.21, samples=4
   iops        : min=77856, max=89610, avg=85730.00, stdev=5359.05, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=4.05%, 1000=13.32%
  lat (msec)   : 2=72.05%, 4=10.41%, 10=0.01%, 20=0.07%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu          : usr=99.30%, sys=0.55%, ctx=2, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=171587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), io=670MiB (703MB), run=2002-2002msec

????方式四雷激、寫(xiě)成我們平常使用的fio形式替蔬，即命令行形式運(yùn)行。和方式三不同的是屎暇，不需要LD_PRELOAD spdk_nvme承桥，可以直接再“ioengine”這里寫(xiě)上spdk_nvme。需注意當(dāng)前filename后面參數(shù)需要加引號(hào)“ --filename='trtype=PCIe traddr=0000.03.00.0 ns=1' ”根悼，filename里面參數(shù)ns可以不寫(xiě)凶异。

root@test-System-Product-Name:/home# /home/fio/fio --ioengine=/home/spdk/build/fio/spdk_nvme --bs=4k --rw=randread --thread=1 --group_reporting=1 --direct=1 --verify=0 --time_based=1 --ramp_time=0 --runtime=2 --iodepth=128 --numjobs=1 --size=1G --name=job --filename='trtype=PCIe traddr=0000.03.00.0'
job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=128
fio-3.35-6-g1b4b
Starting 1 thread
TELEMETRY: No legacy callbacks, legacy socket not created
Jobs: 1 (f=1): [r(1)][60.0%][r=349MiB/s][r=89.3k IOPS][eta 00m:02s]
job: (groupid=0, jobs=1): err= 0: pid=423758: Fri Jun  9 09:48:18 2023
  read: IOPS=71.1k, BW=278MiB/s (291MB/s)(556MiB/2002msec)
    slat (nsec): min=75, max=1398, avg=81.20, stdev= 7.31
    clat (usec): min=93, max=258819, avg=1789.98, stdev=5455.01
     lat (usec): min=93, max=258819, avg=1790.06, stdev=5455.01
    clat percentiles (usec):
     |  1.00th=[   635],  5.00th=[   766], 10.00th=[   873], 20.00th=[  1037],
     | 30.00th=[  1172], 40.00th=[  1303], 50.00th=[  1418], 60.00th=[  1532],
     | 70.00th=[  1647], 80.00th=[  1795], 90.00th=[  2024], 95.00th=[  2212],
     | 99.00th=[  2868], 99.50th=[ 18482], 99.90th=[ 92799], 99.95th=[125305],
     | 99.99th=[141558]
   bw (  KiB/s): min=177896, max=358232, per=100.00%, avg=284250.00, stdev=88507.99, samples=4
   iops        : min=44474, max=89558, avg=71062.50, stdev=22127.00, samples=4
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=4.27%, 1000=13.40%
  lat (msec)   : 2=71.65%, 4=10.04%, 10=0.06%, 20=0.08%, 50=0.08%
  lat (msec)   : 100=0.34%, 250=0.06%, 500=0.01%
  cpu          : usr=99.65%, sys=0.15%, ctx=2, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=142252,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=278MiB/s (291MB/s), 278MiB/s-278MiB/s (291MB/s-291MB/s), io=556MiB (583MB), run=2002-2002msec

～文章已經(jīng)結(jié)束了～

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 人面猴
    序言：七十年代末，一起剝皮案震驚了整個(gè)濱河市挤巡，隨后出現(xiàn)的幾起案子剩彬，更是在濱河造成了極大的恐慌，老刑警劉巖矿卑，帶你破解...
    沈念sama閱讀 217,084評(píng)論 6贊 503
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件喉恋，死亡現(xiàn)場(chǎng)離奇詭異，居然都是意外死亡母廷，警方通過(guò)查閱死者的電腦和手機(jī)轻黑，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 92,623評(píng)論 3贊 392
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進(jìn)店門(mén)，熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)徘意，“玉大人苔悦，你說(shuō)我怎么就攤上這事∽颠郑” “怎么了玖详？”我有些...
    開(kāi)封第一講書(shū)人閱讀 163,450評(píng)論 0贊 353
  • 道士緝兇錄：失蹤的賣(mài)姜人
    文/不壞的土叔 我叫張陵把介，是天一觀的道長(zhǎng)。 經(jīng)常有香客問(wèn)我蟋座，道長(zhǎng)拗踢，這世上最難降的妖魔是什么？ 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 58,322評(píng)論 1贊 293
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任向臀，我火速辦了婚禮巢墅，結(jié)果婚禮上，老公的妹妹穿的比我還像新娘券膀。我一直安慰自己君纫，他們只是感情好，可當(dāng)我...
    茶點(diǎn)故事閱讀 67,370評(píng)論 6贊 390
  • 惡毒庶女頂嫁案：這布局不是一般人想出來(lái)的
    文/花漫 我一把揭開(kāi)白布芹彬。 她就那樣靜靜地躺著蓄髓，像睡著了一般。 火紅的嫁衣襯著肌膚如雪舒帮。 梳的紋絲不亂的頭發(fā)上会喝，一...
    開(kāi)封第一講書(shū)人閱讀 51,274評(píng)論 1贊 300
  • 城市分裂傳說(shuō)
    那天，我揣著相機(jī)與錄音玩郊，去河邊找鬼肢执。 笑死，一個(gè)胖子當(dāng)著我的面吹牛译红，可吹牛的內(nèi)容都是我干的预茄。 我是一名探鬼主播，決...
    沈念sama閱讀 40,126評(píng)論 3贊 418
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開(kāi)眼临庇，長(zhǎng)吁一口氣：“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼反璃！你這毒婦竟也來(lái)了？” 一聲冷哼從身側(cè)響起假夺，我...
    開(kāi)封第一講書(shū)人閱讀 38,980評(píng)論 0贊 275
  • 萬(wàn)榮殺人案實(shí)錄
    序言：老撾萬(wàn)榮一對(duì)情侶失蹤淮蜈，失蹤者是張志新（化名）和其女友劉穎，沒(méi)想到半個(gè)月后已卷，有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體梧田，經(jīng)...
    沈念sama閱讀 45,414評(píng)論 1贊 313
  • ?護(hù)林員之死
    正文 獨(dú)居荒郊野嶺守林人離奇死亡，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 37,599評(píng)論 3贊 334
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年侧蘸，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了裁眯。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
    茶點(diǎn)故事閱讀 39,773評(píng)論 1贊 348
  • 活死人
    序言：一個(gè)原本活蹦亂跳的男人離奇死亡讳癌，死狀恐怖穿稳，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情晌坤，我是刑警寧澤逢艘，帶...
    沈念sama閱讀 35,470評(píng)論 5贊 344
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布旦袋，位于F島的核電站，受9級(jí)特大地震影響它改，放射性物質(zhì)發(fā)生泄漏疤孕。R本人自食惡果不足惜，卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 41,080評(píng)論 3贊 327
  • 男人毒藥：我在死后第九天來(lái)索命
    文/蒙蒙 一央拖、第九天 我趴在偏房一處隱蔽的房頂上張望祭阀。 院中可真熱鬧，春花似錦鲜戒、人聲如沸专控。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 31,713評(píng)論 0贊 22
  • 一樁弒父案袍啡，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)踩官。三九已至，卻和暖如春境输，著一層夾襖步出監(jiān)牢的瞬間，已是汗流浹背颖系。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 32,852評(píng)論 1贊 269
  • 情欲美人皮
    我被黑心中介騙來(lái)泰國(guó)打工嗅剖， 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留，地道東北人嘁扼。 一個(gè)月前我還...
    沈念sama閱讀 47,865評(píng)論 2贊 370
  • 代替公主和親
    正文 我出身青樓信粮，卻偏偏與公主長(zhǎng)得像，于是被迫代替她去往敵國(guó)和親趁啸。 傳聞我的和親對(duì)象是個(gè)殘疾皇子强缘，可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 44,689評(píng)論 2贊 354

推薦閱讀更多精彩內(nèi)容