云原生現(xiàn)場(chǎng)分析 Part4 —— eBPF 跟蹤 Istio/Envoy 之 upstream/downstream 事件驅(qū)動(dòng)協(xié)作下的 HTTP 反向代理流程

image.png

注婶博,原文來(lái)自 https://blog.mygraphql.com/zh/posts/low-tec/trace/trace-istio/trace-istio-part4/ 瓮具。如你看到的轉(zhuǎn)載圖片不清,請(qǐng)回到原文凡人。

為何要了解 upstream/downstream 事件驅(qū)動(dòng)協(xié)作下的 HTTP 反向代理流程

想不到還真堅(jiān)持寫(xiě)到 Part4 了名党。不管你信不信,每一 part 的“為何”一節(jié)划栓,是我覺(jué)得最難寫(xiě)的 :) 。如果你是第一次讀這個(gè)系列条获,不用擔(dān)心忠荞,每一節(jié)都是比較獨(dú)立的。

不出意外的話帅掘,你之前看到的 Envoy 簡(jiǎn)介是這樣描述 Envoy 的特性的:

  • C++ 編寫(xiě)委煤,原生底層,沒(méi)有 GC stop the world修档,所以性能優(yōu)異
  • 異步事件驅(qū)動(dòng)碧绞,多路復(fù)用,完美解決 C10k problem
  • 由于單線程負(fù)責(zé)多連接吱窝,在大連接量時(shí)讥邻,減少了大量線程的內(nèi)存開(kāi)銷(xiāo),和 CPU 上下文切換開(kāi)銷(xiāo)

這些描述院峡,當(dāng)然有其合理性兴使。但很多遠(yuǎn)看很美麗的事物,在微距放大后照激,可能有很多有意思发魄,有價(jià)值的東西。我相信只要分析得足夠細(xì)致,總可能針對(duì)我們實(shí)際運(yùn)行環(huán)境和流量特性對(duì)作一些有意義的優(yōu)化励幼√ⅲ可能只是修改一個(gè) Envoy / Kernel 的配置,也可能是修改一行 Envoy 的代碼苹粟∮谢或你的應(yīng)用的流程相關(guān)行為,如每次寫(xiě) socket 的 buffer 數(shù)據(jù)包的大小六水。

而這些俺孙,都需要建立在了解實(shí)現(xiàn)細(xì)節(jié)的基礎(chǔ)上。除非覺(jué)得運(yùn)氣或經(jīng)驗(yàn)值特別好掷贾,可以瞎猜睛榄。

逆向工程與云原生現(xiàn)場(chǎng)分析 系列介紹

開(kāi)始前先做個(gè)預(yù)告,逆向工程與云原生現(xiàn)場(chǎng)分析 系列(將)包括:

在系列中港准,我將演示如何讓 bpftrace "讀" 懂運(yùn)行期的由 C++ 11 編寫(xiě)成的 envoy 進(jìn)程中的對(duì)象數(shù)據(jù)旨剥。
為免嚇跑人,還是老套路浅缸,多圖少代碼轨帜,不過(guò)有的圖有點(diǎn)點(diǎn)復(fù)雜。程序員大叔開(kāi)始講故事了衩椒。??

HTTP 反向代理的總流程

整體看栗涂,Socket 事件驅(qū)動(dòng)的 HTTP 反向代理總流程如下:


image.png

圖中看出填物,有4種事件驅(qū)動(dòng)了整個(gè)流程竿痰。后面幾節(jié)會(huì)逐個(gè)分析褪测。

為免一下子進(jìn)入各個(gè)步驟細(xì)節(jié)而讓人迷途,現(xiàn)在先看看所有步驟的總流程:

image.png

圖:Istio/Envoy 模塊協(xié)作總圖

Downstream Read Request 模塊協(xié)作

image.png

圖:Downstream Read-Ready 模塊協(xié)作

大概說(shuō)明一下流程:

  1. upstream socket 可讀回調(diào)
  2. Http::ConnectionManagerImpl 讀取 socket阁将,增量放入 Http1::ConnectionImpl
  3. Http1::ConnectionImpl 調(diào)用 nghttp2 增量解釋 HTTP 請(qǐng)求
  4. 如果 nghttp2 認(rèn)為已經(jīng) 完整讀取了 HTTP Request 請(qǐng)求膏秫,則調(diào)用 Http::ServerConnection::onMessageCompleteBase()
  5. Http::ServerConnection::onMessageCompleteBase() 首先 停止 downstream ReadReady 監(jiān)聽(tīng)
  6. Http::ServerConnection::onMessageCompleteBase() 調(diào)用 Http::FilterManager ,發(fā)起 http filter chain 的 decodeHeaders 迭代流程
  7. 一般做盅,http filter chain 的最后一個(gè) http filter 是 Router::Filter 缤削,Router::Filter::decodeHeaders() 被調(diào)用
  8. Router::Filter::decodeHeaders() 的邏輯就見(jiàn)下圖了。
image.png

圖:Downstream Request Router 模塊協(xié)作

大概說(shuō)明一下流程:

  1. Router::Filter 吹榴,Router::Filter::decodeHeaders() 被調(diào)用
  2. 根據(jù)配置的 Router 規(guī)則僻他,匹配到 Cluster
  3. 如 Cluster 連接池對(duì)象不存在,則新建
  4. 新建 Envoy::Router::UpstreamRequest 對(duì)象腊尚。
  5. 調(diào)用 Envoy::Router::UpstreamRequest::encodeHeaders(bool end_stream) 吨拗, encode HTTP header
  6. 經(jīng)過(guò)一系列的負(fù)載均衡算法,匹配到 upstream 的 host(endpoint)
  7. 發(fā)現(xiàn)到選定的 upstream host 的連接不足,則:
    1. 打開(kāi)一新的 socket fd(未連接)
    2. 注冊(cè) upstream socket FD 的 WriteReady / Connected 事件劝篷。 準(zhǔn)備在事件回調(diào)時(shí)寫(xiě) upstream request
    3. 用 socket fd 發(fā)起到 upstream host 的異步連接請(qǐng)求
  8. 關(guān)聯(lián) downstream 與 upstream fd

Upstream Write Request 模塊協(xié)作

[圖片上傳失敗...(image-5b18c8-1649291572197)]
大概說(shuō)明一下流程:

  1. upstream socket write ready 回調(diào)
  2. 發(fā)現(xiàn)是連接成功回調(diào)哨鸭,關(guān)聯(lián) upstream socket 到 ConnectionPool::ActiveClient
  3. upstream socket write ready 回調(diào)
  4. 發(fā)現(xiàn)是連接可寫(xiě),寫(xiě)入 upstream HTTP request

Upstream Read Response 模塊協(xié)作

image.png

Downstream Write Response 模塊協(xié)作

image.png

BPF 腳本輸出

以上圖片娇妓,不是只讀源碼就得出的像鸡。是需要通過(guò) bpf 埋點(diǎn)觀察:

  1. 記錄 downstream FD(socket 的 file descriptor),可以認(rèn)為是進(jìn)程中的 socket id
  2. 埋點(diǎn)進(jìn)程級(jí)和 kernel 級(jí)的相關(guān)函數(shù)入口和出口哈恰,觀察其出入?yún)⑴c stack
  3. 關(guān)聯(lián)起 downstream FD 與 upstream FD

其中當(dāng)然有很多細(xì)節(jié)只估,不過(guò)我不打算一個(gè)個(gè)講了,實(shí)在有需要用着绷,想深入了解的同學(xué)可以聯(lián)系我探討蛔钙。下一節(jié)的 bpf 代碼是最好的細(xì)節(jié)。

https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.3.log

#### Downstream Read Request: accept downstream socket荠医, 可見(jiàn) downstream socket 的 FD=41

OS handshaked TCP:
01:22:11 4215   wrk:worker_0   172.30.207.129                          58708 172.21.206.232                          15006 0/4096
sys_exit_accept4 fd=41

        accept4+96
        Envoy::Network::IoSocketHandleImpl::accept(sockaddr*, unsigned int*)+82
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+216
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217
 

***** elapsed=1621361178: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__110__function6__funcIZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS2_7Network21ListenerFilterMatcherEEERNS2_6Server13Configuration22ListenerFactoryContextEEUlRNSD_21ListenerFilterManagerEE_NS_9allocatorISO_EEFvSN_EEclESN_

***** elapsed=1621785490: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS3_7Network21ListenerFilterMatcherEEERNS3_6Server13Configuration22ListenerFactoryContextEEUlRNSE_21ListenerFilterManagerEE_SO_EEEvDpOT_

***** elapsed=1630898092: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC1ENSt3__110shared_ptrINS2_6ConfigEEE

***** elapsed=1630908037: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC2ENSt3__110shared_ptrINS2_6ConfigEEE

***** elapsed=1631036930: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter8onAcceptERNS_7Network23ListenerFilterCallbacksE

***** elapsed=1631045047: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter6onReadEv

***** elapsed=1631070582: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_recvfrom, fd=41, ret=89

        recv+108


***** elapsed=1631085065: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter16parseClientHelloEPKvm

#### Downstream Read Request: 選擇 network filter chain
***** elapsed=1631145371: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterD0Ev
comm:wrk:worker_0,tid:5327: Got setFilterChainName=0.0.0.0_8080, lenght=12

        Envoy::StreamInfo::StreamInfoImpl::setFilterChainName(absl::string_view)+0
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107

#### Downstream Read Request: 監(jiān)聽(tīng) upstream FD 事件
***** elapsed=1631199672: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
        Envoy::Network::ServerConnectionImpl::ServerConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+107
        Envoy::Event::DispatcherImpl::createServerConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&)+70
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+307
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl9addFilterENSt3__110shared_ptrINS0_6FilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+0
        std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
        virtual thunk to Envoy::Network::ConnectionImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+62
        std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
        std::__1::__function::__func<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7, std::__1::allocator<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+353
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

***** elapsed=1631443267: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1631456738: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=31393726, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=100 
***************************

#### Downstream Read Request: 讀 HTTP Request
***** elapsed=1631475016: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1631513821: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=89

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1631527775: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=-11

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl6onReadEv,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
***** elapsed=1631671079: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005

#### Downstream Read Request: downstream 的 HTTP Reuest 已經(jīng)完全讀扔跬选(onMessageComplete())。停止監(jiān)聽(tīng) downstream 的 ReadReady 事件
***** elapsed=1631678652: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80002004
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Network::ConnectionImpl::readDisable(bool)+1077
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+86
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

#### Downstream Read Request: 監(jiān)聽(tīng)新建立的 upstream socket 的 FD=44 彬向, 這里 socket 還處于未連接狀態(tài)〖婀保現(xiàn)在監(jiān)聽(tīng) upstream socket 的 所有事件。以備后面異步連接成功時(shí)收到回調(diào)
***** elapsed=1632388633: tid=5327,comm=wrk:worker_0: register upstream event trigger:sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=44, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
        Envoy::Network::ClientConnectionImpl::ClientConnectionImpl(Envoy::Event::Dispatcher&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+283
        Envoy::Event::DispatcherImpl::createClientConnection(std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+75
        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterInfo const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, Envoy::Network::TransportSocketFactory&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>)+561
        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+126
        non-virtual thunk to Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+21
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+112
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98f08478,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
        Envoy::Http::CodecClient::CodecClient(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&)+357
        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+70
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0    : setsockopt: level=6, fd=44, optname=1, optval=1, optlen=4. 

#### Downstream Read Request: 發(fā)現(xiàn)異步 upstream 連接
connecting to 8080:
01:23:36 4215     wrk:worker_0     127.0.0.6                               36383  172.21.206.232                          8080  

        connect+75
        Envoy::Network::IoSocketHandleImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+104
        Envoy::Network::SocketImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+64
        Envoy::Network::ClientConnectionImpl::connect()+650
        Envoy::Http::CodecClient::connect()+543
        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+694
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

***** elapsed=1632608064: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1632618881: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1145219, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=92 
***************************


***** elapsed=1632635526: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1632654517: tid=5327,comm=wrk:worker_0: END:EventFired

#### Upstream Write Request: 異步 upstream 連接成功娃胆,回調(diào)遍希。
***** elapsed=1632659321: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET

comm:wrk:worker_0,tid:5327: ConnPoolImplBase*attachStreamToClient: uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy14ConnectionPool16ConnPoolImplBase20attachStreamToClientERNS0_12ActiveClientERNS0_13AttachContextE 

        Envoy::ConnectionPool::ConnPoolImplBase::attachStreamToClient(Envoy::ConnectionPool::ActiveClient&, Envoy::ConnectionPool::AttachContext&)+0
        Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent(Envoy::ConnectionPool::ActiveClient&, absl::string_view, Envoy::Network::ConnectionEvent)+497
        Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent)+59
        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+511
        Envoy::Network::ConnectionImpl::onWriteReady()+1648
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

#### Upstream Write Request: 向 upstream socket 寫(xiě) HTTP Request
***** elapsed=1632741863: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=44, ret=262

        writev+77
        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
        Envoy::Network::ConnectionImpl::onWriteReady()+1876
        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+529
        Envoy::Network::ConnectionImpl::onWriteReady()+1648
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1632756069: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1632764448: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=129648, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=60 
***************************

#### Upstream Read Response: upstream socket ReadReady 回調(diào)
***** elapsed=1637821708: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET

#### Upstream Read Response: 讀 upstream HTTP Response
***** elapsed=1637898855: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=4511

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1637915998: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=-11

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

***** elapsed=1638220184: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80002004

#### Upstream Read Response: 已經(jīng)完整讀取 upstream HTTP Response,開(kāi)始監(jiān)聽(tīng) downstream 的 WriteReady 事件里烦,準(zhǔn)備寫(xiě) downstream HTTP Response
***** elapsed=1638231319: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Network::ConnectionImpl::readDisable(bool)+938
        Envoy::Http::Http1::StreamEncoderImpl::~StreamEncoderImpl()+112
        non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onEncodeComplete()+54
        Envoy::Http::Http1::StreamEncoderImpl::endEncode()+166
        Envoy::Http::Http1::StreamEncoderImpl::encodeData(Envoy::Buffer::Instance&, bool)+340
        Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData(Envoy::Buffer::Instance&, bool)+679
        Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState)+2138
        Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase()+619
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7705
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        Envoy::Http::Http1::ClientConnectionImpl::dispatch(Envoy::Buffer::Instance&)+29
        Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48
        Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92


***** elapsed=1638898463: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1638934709: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1123611, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
***************************


***** elapsed=1638955421: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1638975366: tid=5327,comm=wrk:worker_0: END:EventFired

#### Downstream Write Response: downstream 的 WriteReady 事件回調(diào)凿蒜。
***** elapsed=1638980295: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET

#### Downstream Write Response: 寫(xiě) downstream HTTP Response
***** elapsed=1639253752: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=41, ret=4631

        writev+77
        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
        Envoy::Network::ConnectionImpl::onWriteReady()+1876
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1639293010: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1639308346: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=354486, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
***************************


***** elapsed=1643785802: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1643855427: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=0

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

***** elapsed=1643880789: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
sys_enter_close fd=41
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98d38758
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98996380

***** elapsed=1644006858: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1644056452: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=289169, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=52 
***************************

^C


@epoll_ctl_op[1]: EPOLL_CTL_ADD
@epoll_ctl_op[2]: EPOLL_CTL_DEL
@epoll_ctl_op[3]: EPOLL_CTL_MOD


@fd2sockopt[44, 6, 1, 1]: 1
@fd2sockopt[41, 6, 1, 1]: 1


@filterManagerImpl2fd[94143954060408]: 41
@filterManagerImpl2fd[94143950909952]: 41

BPF 腳本

https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.bt

#!usr/bin/bpftrace
/*
IMPORT-ENV: $ENVOY_PID
args: $1=ENVOY_PID
*/

/*

#export PODID=`sudo crictl  pods | grep fortio-server | egrep '\bReady' | awk '{print $1}'`
#export CONTAINERID=`sudo crictl ps | grep $PODID | grep istio-proxy |  awk '{print $1}'`

kubectl -n idm-mark exec -it fortio-server -c  istio-proxy  -- bash -c 'ls -l /proc/`pgrep envoy`/ns/pid'
lrwxrwxrwx 1 istio-proxy istio-proxy 0 Mar 31 09:18 /proc/15/ns/pid -> 'pid:[4026532799]'
pgrep envoy | tee /dev/fd/2 | xargs -L 1 -I '{}' sudo ls -l /proc/'{}'/ns 

export PID=4215
export SCRIPT_HOME=`pwd`
export bpftrace_image=cndt-bcc-ub

docker run -it --rm --init  --privileged --name bpftrace -h bpftrace \
    --pid host \
    --net host \
    -e SCRIPT_HOME=$SCRIPT_HOME \
    -e PID=$PID \
    -e ENVOY_PID=$PID \
    -e BT=trace-envoy-filter-router.bt \
    -v /etc/localtime:/etc/localtime:ro \
    -v /sys:/sys:rw \
    -v /usr/src:/usr/src:rw \
    -v /lib/modules:/lib/modules:ro \
    -v ${SCRIPT_HOME}:${SCRIPT_HOME}:rw \
    $bpftrace_image \
    ${SCRIPT_HOME}/warp-bt.sh $PID
*/


/*
1. BPF Map 說(shuō)明:

1.1. range tid map
- @watchedWakeRound[tid]=tid
       - sys_exit_accept4
       - FileEventImpl*assignEvents*
- @fdFired[tid]=$fd;
       - uprobe: *FileEventImpl*assignEvents*
       - uretprobe: *FileEventImpl*assignEvents*

1.2. fd map
- @fd2sockpair[$fd]=@sockpair[tid]
       - sys_exit_accept4
       - sys_enter_close
- @fd2sockopt[$fd, $level, $optname, $optval_int] = 1
       - sys_enter_setsockopt

- @filterManagerImpl2fd[arg0] = @fdFired[tid] // filterManagerImpl to downstream fd
- @fd2filterManagerImpl[$fd] = arg0 
- @fd2filterManagerImpl2[$fd] = arg0
- @downFD2upFD[$downFD] = $fd; //downstream fd to upstream fd
       - sys_enter_epoll_ctl
- @upFD[$fd] = $fd; //upstream fd list
       - sys_enter_epoll_ctl

1.3. temp tid map
- @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
- @sockpair_exist[tid]=1
- @watchedIo[tid]=fd

*/

#include <linux/in.h>
#include <linux/in6.h>
#include <linux/socket.h>
#include <net/sock.h>
// #include <sys/epoll.h>

// The event argument describes the object linked to the file descriptor fd.  The struct epoll_event is defined as:
// struct epoll_data_t {
//     int          fd;
// };

// struct epoll_event {
//     //a bit mask composed by ORing together zero or more of the following available event types
//     uint32_t     events;      /* Epoll events */

//     struct epoll_data_t data;        /* User data variable */
// };



BEGIN
{
    printf("Tracing Envoy. Hit Ctrl-C to end.\n");
       printf("#define EPOLL_CTL_ADD 1\n #define EPOLL_CTL_DEL 2\n #define EPOLL_CTL_MOD 3\n");

       @epoll_ctl_op[1]="EPOLL_CTL_ADD";
       @epoll_ctl_op[2]="EPOLL_CTL_DEL";
       @epoll_ctl_op[3]="EPOLL_CTL_MOD";


}

/*
獲取 accept 連接時(shí)的本地和對(duì)端地址。并標(biāo)記本輪 wakeup 需要監(jiān)控
 */
kretprobe:inet_csk_accept
/pid==$1 /
{
    $sk = (struct sock *)retval;
    $inet_family = $sk->__sk_common.skc_family;

    if ($inet_family == AF_INET || $inet_family == AF_INET6) {
        // initialize variable type:
        $daddr = ntop(0);
        $saddr = ntop(0);
        if ($inet_family == AF_INET) {
            $daddr = ntop($sk->__sk_common.skc_daddr);
            $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
        } else {
            printf("not support IPv6.\n");
            return;
        }
        
        $lport = $sk->__sk_common.skc_num;

              // printf("accept(), port=%d\n", $lport);

              //only watch listen port 15006
        if( 15006 != $lport ) { 
            return;
        }

        $dport = $sk->__sk_common.skc_dport;
        $qlen  = $sk->sk_ack_backlog;
        $qmax  = $sk->sk_max_ack_backlog;

        // Destination port is big endian, it must be flipped
        $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);

              printf("OS handshaked TCP:\n");
        time("%H:%M:%S ");
        printf("%-6d %-14s ", pid, comm);
        printf("%-39s %-5d %-39s %-5d ", $daddr, $dport, $saddr,
            $lport);
        printf("%d/%d\n", $qlen, $qmax);

        @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
        @sockpair_exist[tid]=1;

              @watchedWakeRound[tid]=tid;
    }
}

/**
 * 獲取新連接的 FD 招驴。并標(biāo)記本輪 libevent 回調(diào)需要監(jiān)控
 */
tracepoint:syscalls:sys_exit_accept4
/pid==$1 && @sockpair_exist[tid] /
{
    $fd = args->ret;
    if( $fd < 0 ) {
        return;
    }
    printf("sys_exit_accept4 fd=%d\n", $fd);
    @fd2sockpair[$fd]=@sockpair[tid];

       @fdFired[tid]=$fd;

    delete(@sockpair[tid]);
    delete(@sockpair_exist[tid]);
       printf("%s \n", ustack());
}

/**
 * 記錄 FD 的 sockopt
 */
tracepoint:syscalls:sys_enter_setsockopt
/pid==$1/
{
       // socket opts: https://elixir.bootlin.com/linux/v5.16.3/source/include/uapi/linux/tcp.h#L92     
       $level = args->level;
       $fd = args->fd;

       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
              $optname = args->optname;
              $optval = args->optval;
              $optval_int = *$optval;
              $optlen = args->optlen;
              // printf("\n########## setsockopt() ##########\n");
              printf("comm:%-16s: setsockopt: level=%d, fd=%d, optname=%d, optval=%d, optlen=%d. \n", comm, $level, $fd, $optname, $optval_int, $optlen);
              @fd2sockopt[$fd, $level, $optname, $optval_int] = 1;
       }
}



/*
記錄 epoll 監(jiān)聽(tīng)的 FD

cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_ctl/format
 */
tracepoint:syscalls:sys_enter_epoll_ctl
/pid==$1/
{
       $fd=args->fd;
       $epollEvents = (uint32) (args->event->events);
       $op = args->op;
       if( @fd2sockpair[$fd].0 ) {

              printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                     elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);


              if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                     if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                            printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                     }
                     if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                            printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                     }
                     if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                            printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                     }
                     printf("%s\n", ustack(30));
              }
              
       } else {
              $downFD = @fdFired[tid];
              if( $downFD ) {//in downstream fd event callback, register upstream event trigger
                     @upFD[$fd] = $fd;
                     @downFD2upFD[$downFD] = $fd;

                     printf("***** elapsed=%d: tid=%d,comm=%s: register upstream event trigger:sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                            elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);


                     if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                            if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                            }
                            if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                            }
                            if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                            }
                            printf("%s\n", ustack(60));
                     }                     
              }
       }
}

/*
結(jié)束本輪的 wakeRound/runnableRound篙程,并等待下一輪
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_wait/format
 */
tracepoint:syscalls:sys_enter_epoll_wait
/pid==$1/
{

       if( @watchedWakeRound[tid] ) {
              $runnableStartTime=@tid2Waketime[tid];
              if( $runnableStartTime ) {
                     $runableDuaration = elapsed - $runnableStartTime;
                     printf("\n******* WAKE-ROUND:END Summary *******\n");
                     printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait, runableDuaration=%d, tid2epollNrFdReady=%d\n", 
                            elapsed, tid, comm, $runableDuaration, @tid2epollNrFdReady[tid]);
                     $tid_last_epoll_wait_args = @last_epoll_wait_args[tid];
                     if( $tid_last_epoll_wait_args.0  ) {
                            printf("*** last_epoll_wait_args: epfd=%d, events=%d, maxevents=%d, timeout=%d \n", 
                                   $tid_last_epoll_wait_args.0, $tid_last_epoll_wait_args.1, $tid_last_epoll_wait_args.2, $tid_last_epoll_wait_args.3);
                     }
                     printf("***************************\n\n");
              }
       }

       delete(@tid2Waketime[tid]);
       delete(@watchedWakeRound[tid]);
       delete(@tid2epollNrFdReady[tid]);

       //read in next sys_enter_epoll_wait
       @last_epoll_wait_args[tid]=(args->epfd, args->events, args->maxevents, args->timeout);
}

/*
開(kāi)始本輪 wakeRound/runnableRound
cat /sys/kernel/debug/tracing/events/syscalls/sys_exit_epoll_wait/format
 */
tracepoint:syscalls:sys_exit_epoll_wait
/pid==$1/
{
       // printf("\n***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait\n", elapsed, tid, comm);

       // printf("epfd: 0x%08lx, events: 0x%08lx, maxevents: 0x%08lx, timeout: 0x%08lx \n", 
       //        ((args->epfd)), ((args->events)), ((args->maxevents)), ((args->timeout)) );

       @tid2Waketime[tid]=elapsed;
       @tid2epollNrFdReady[tid]=args->ret;
}

/*
記錄本輪 epoll 事件觸發(fā)的 FD 級(jí)回調(diào)開(kāi)始

Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int fd, short events, void* fileEventImplThis)  !!!NOT!!: FileEventImpl::assignEvents()
C++11 Lambda expressions:
event_assign( &raw_event_, base, fd_, xxx, [](evutil_socket_t, short what, void* arg)-> void {}, this );

event_assign(): https://libevent.org/doc/event_8h.html#a3e49a8172e00ae82959dfe64684eda11
       event_assign (   struct event *  ev,
              struct event_base *   base,
              evutil_socket_t   fd,
              short     events,
              event_callback_fn     callback,
              void *    callback_arg 
       )    

       https://libevent.org/doc/event_8h.html#aed2307f3d9b38e07cc10c2607322d758
       typedef void(* event_callback_fn) (evutil_socket_t, short, void *)
              fd    An fd or signal
              events    One or more EV_* flags
              arg   A user-supplied argument.
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1/ 
{
       $fd = arg0;
       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
              $libevent_events = arg1;

              printf("\n***** elapsed=%d: tid=%d,comm=%s: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()\n", elapsed, tid, comm);
              printf("FileEventImpl*=%p, fd=%d, events=0x%x\n",arg2, $fd, $libevent_events);

              if( $libevent_events & (uint16)0x01 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_TIMEOUT\n");
              }
              if( $libevent_events & (uint16)0x02 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_READ\n");
              }
              if( $libevent_events & (uint16)0x04 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_WRITE\n");
              }
              if( $libevent_events & (uint16)0x20 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_ET\n");
              }
              if( $libevent_events & (uint16)0x80 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_CLOSED\n");
              }
              
              printf("%s\n", kstack);
              @fdFired[tid]=$fd;
              @watchedWakeRound[tid]=tid;
       }
}

/*
記錄本輪 epoll 事件觸發(fā)的 FD 級(jí)回調(diào)結(jié)束
*/
uretprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1 && @fdFired[tid] / 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: END:EventFired\n", elapsed, tid, comm);
       delete(@fdFired[tid]);
}

/*
埋點(diǎn) socket 讀
*/ 
tracepoint:syscalls:sys_enter_read,tracepoint:syscalls:sys_enter_readv,tracepoint:syscalls:sys_enter_recv*
/pid == $1 && (@fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
{
       @watchedIo[tid]=args->fd;
}

tracepoint:syscalls:sys_exit_read,tracepoint:syscalls:sys_exit_readv,tracepoint:syscalls:sys_exit_recv*
/pid == $1 && @watchedIo[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_read, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid],args->ret);
       delete(@watchedIo[tid]);
       printf("%s\n", ustack(20));
}

/*
埋點(diǎn) socket 寫(xiě)
*/
tracepoint:syscalls:sys_enter_write,tracepoint:syscalls:sys_enter_writev
/pid == $1 && ( @fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
{
       @watchedIo[tid]=args->fd;
}

tracepoint:syscalls:sys_exit_write,tracepoint:syscalls:sys_exit_writev
/pid == $1 && @watchedIo[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_write, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid], args->ret);
       delete(@watchedIo[tid]);
       printf("%s\n", ustack(20));
}


/*
打印 TlsInspector 的所有函數(shù)調(diào)用
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*TlsInspector*
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: TlsInspector*, probe=%s\n", elapsed, tid, comm, probe);
}

/*
打印連接確認(rèn)協(xié)后的函數(shù)調(diào)用
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnectionSocketImpl*setRequestedApplicationProtocols*
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: ConnectionSocketImpl::setRequestedApplicationProtocols\n", elapsed, tid, comm);
       // printf("%s", ustack);
}

//SslSocket::SslSocket
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocketC2ENSt3__110shared_ptrINS_3Ssl7ContextEEENS2_12InitialStateERKNS5_IKNS_7Network22TransportSocketOptionsEEENS4_8functionIFNS5_INS6_10HandshakerEEENS4_10unique_ptrI6ssl_stN4bssl8internal7DeleterISK_EEEEiPNS6_18HandshakeCallbacksEEEE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// setTransportSocketCallbacks
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket27setTransportSocketCallbacksERNS_7Network24TransportSocketCallbacksE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}


// SslSocket::doRead
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket6doReadERNS_6Buffer8InstanceE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}


// SslSocket::doWrite
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket7doWriteERNS_6Buffer8InstanceEb
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::onSuccess
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_N5Envoy10Extensions16TransportSockets3Tls9SslSocket9onSuccessEP6ssl_st
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

/*
打印匹配到的 Network Fitler Chain 名字
void setFilterChainName(absl::string_view filter_chain_name)
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10StreamInfo14StreamInfoImpl18setFilterChainNameEN4absl11string_viewE 
/pid == $1 && @fdFired[tid]/ 
{ 
/**
[Assembly 2: Calling convention](https://cs61.seas.harvard.edu/site/2018/Asm2/)

1. A structure argument that fits in a single machine word (64 bits/8 bytes) is passed in a single register.
   
    Example: `struct small { char a1, a2; }`

2. A structure that fits in two to four machine words (16–32 bytes) is passed in sequential registers, as if it were multiple arguments.
   
    Example: `struct medium { long a1, a2; }`
*/
    $filterName = str(reg("si"));
    $filterNameLength = reg("dx");

    // printf("tid:%d: Got setFilterChainName=%s, lenght=%d\n %s \n", tid, $filterName, $filterNameLength, ustack ); 
    printf("comm:%s,tid:%d: Got setFilterChainName=%s, lenght=%d\n", comm, tid, $filterName, $filterNameLength ); 

    if( $filterNameLength > 0 ) {
        printf("%s\n", ustack(3));
    }
}

/*
埋點(diǎn) FilterManagerImpl::add*Filter()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*add*Filter*,uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*FilterManagerImpl*
/pid == $1 && @fdFired[tid]/ 
{ 
       if( @upFD[@fdFired[tid]] ) {
              return;
       }

       $fd = @fdFired[tid];
       @filterManagerImpl2fd[arg0] = $fd;
       if( @fd2filterManagerImpl[$fd] ) {
              @fd2filterManagerImpl2[$fd] = arg0;
       } else {
              @fd2filterManagerImpl[$fd] = arg0;
       }
       printf("comm:%s,tid:%d: FilterManagerImpl.add*Filter|FilterManagerImpl::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd);
       printf("%s\n", ustack(40));
}

/*
埋點(diǎn) FilterManagerImpl::on*()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*on*
/pid == $1 && @fdFired[tid]/ 
{ 
       if( @upFD[@fdFired[tid]] ) {
              return;
       }

       $fd = @fdFired[tid];
       @filterManagerImpl2fd[arg0] = $fd;
       if( @fd2filterManagerImpl[$fd] ) {
              @fd2filterManagerImpl2[$fd] = arg0;
       } else {
              @fd2filterManagerImpl[$fd] = arg0;
       }
       printf("comm:%s,tid:%d: FilterManagerImpl.on*::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd); 
}

/*
埋點(diǎn) ConnPoolImplBase::attachStreamToClient()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnPoolImplBase*attachStreamToClient*
/pid == $1 && @fdFired[tid]/ 
{ 
       printf("comm:%s,tid:%d: ConnPoolImplBase*attachStreamToClient: %s \n", comm, tid, probe); 
       printf("%s\n", ustack(40));
}

/*
埋點(diǎn) socket connect upstream
*/
kprobe:tcp_connect
/pid == $1/
{
       $sk = ((struct sock *)arg0);
       $inet_family = $sk->__sk_common.skc_family;

       if ($inet_family == AF_INET || $inet_family == AF_INET6)
       {
              if ($inet_family == AF_INET)
              {
                     $daddr = ntop($sk->__sk_common.skc_daddr);
                     $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
              }
              else
              {
                     $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
                     $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
              }
              $lport = $sk->__sk_common.skc_num;
              $dport = $sk->__sk_common.skc_dport;

              // Destination port is big endian, it must be flipped
              $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);

              // only watch listen port 8080
              if (8080 != $dport)
              {
                     return;
              }

              printf("connecting to 8080:\n");
              time("%H:%M:%S ");
              printf("%-8d %-16s ", pid, comm);
              printf("%-39s %-6d %-39s %-6d\n", $saddr, $lport, $daddr, $dport);
              printf("%s\n", ustack(40));
       }
}


/*
清理關(guān)閉的 FD 相關(guān)的 Map
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_close/format
*/
tracepoint:syscalls:sys_enter_close
/pid==$1/
{
    $fd = args->fd;
    if( $fd < 0 ) {
        return;
    }

       delete(@upFD[$fd]);

       if( @fd2sockpair[$fd].0 ) {
              printf("sys_enter_close fd=%d\n", $fd);
       }
       else {
              return;
       }
    delete(@fd2sockpair[$fd]);

       /////

       $filterManagerImpl = @fd2filterManagerImpl[$fd];
       if( $filterManagerImpl ) {
              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
              delete( @fd2filterManagerImpl[$fd] );
       }

       if( @filterManagerImpl2fd[$filterManagerImpl] ) {
              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
              delete(@filterManagerImpl2fd[$filterManagerImpl]);
       }


       $filterManagerImpl = @fd2filterManagerImpl2[$fd];
       if( $filterManagerImpl ) {
              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
              delete( @fd2filterManagerImpl2[$fd] );
       }

       if( @filterManagerImpl2fd[$filterManagerImpl] ) {
              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
              delete(@filterManagerImpl2fd[$filterManagerImpl]);
       }

       delete(@upFD[@downFD2upFD[$fd]]);
       delete(@downFD2upFD[$fd]);
}

END
{
       clear(@last_epoll_wait_args);
       clear(@tid2Waketime);
       clear(@tid2epollNrFdReady);
}

結(jié)尾

這 part枷畏,從 socket 事件驅(qū)動(dòng)的角度别厘,了解了 envoy 作為反向代理的主要流程。我自己覺(jué)得是學(xué)習(xí)到東西的拥诡,你呢触趴?
寫(xiě)博客的初憧是記錄自己。想不到也能通過(guò)博客認(rèn)識(shí)到很多志同道合渴肉,對(duì)技術(shù)認(rèn)真之人(不能叫同志冗懦?)。謝謝你們仇祭,我才能堅(jiān)持寫(xiě)下去披蕉,也認(rèn)為這樣的寫(xiě)作有一定價(jià)值!

image.png

這是幾年前拍的老照片了,共勉没讲。

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 序言:七十年代末眯娱,一起剝皮案震驚了整個(gè)濱河市,隨后出現(xiàn)的幾起案子爬凑,更是在濱河造成了極大的恐慌徙缴,老刑警劉巖,帶你破解...
    沈念sama閱讀 216,591評(píng)論 6 501
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件嘁信,死亡現(xiàn)場(chǎng)離奇詭異于样,居然都是意外死亡,警方通過(guò)查閱死者的電腦和手機(jī)潘靖,發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 92,448評(píng)論 3 392
  • 文/潘曉璐 我一進(jìn)店門(mén)穿剖,熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái),“玉大人秘豹,你說(shuō)我怎么就攤上這事携御。” “怎么了既绕?”我有些...
    開(kāi)封第一講書(shū)人閱讀 162,823評(píng)論 0 353
  • 文/不壞的土叔 我叫張陵啄刹,是天一觀的道長(zhǎng)。 經(jīng)常有香客問(wèn)我凄贩,道長(zhǎng)誓军,這世上最難降的妖魔是什么? 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 58,204評(píng)論 1 292
  • 正文 為了忘掉前任疲扎,我火速辦了婚禮昵时,結(jié)果婚禮上,老公的妹妹穿的比我還像新娘椒丧。我一直安慰自己壹甥,他們只是感情好,可當(dāng)我...
    茶點(diǎn)故事閱讀 67,228評(píng)論 6 388
  • 文/花漫 我一把揭開(kāi)白布壶熏。 她就那樣靜靜地躺著句柠,像睡著了一般。 火紅的嫁衣襯著肌膚如雪棒假。 梳的紋絲不亂的頭發(fā)上溯职,一...
    開(kāi)封第一講書(shū)人閱讀 51,190評(píng)論 1 299
  • 那天,我揣著相機(jī)與錄音帽哑,去河邊找鬼谜酒。 笑死,一個(gè)胖子當(dāng)著我的面吹牛妻枕,可吹牛的內(nèi)容都是我干的僻族。 我是一名探鬼主播粘驰,決...
    沈念sama閱讀 40,078評(píng)論 3 418
  • 文/蒼蘭香墨 我猛地睜開(kāi)眼,長(zhǎng)吁一口氣:“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼述么!你這毒婦竟也來(lái)了晴氨?” 一聲冷哼從身側(cè)響起,我...
    開(kāi)封第一講書(shū)人閱讀 38,923評(píng)論 0 274
  • 序言:老撾萬(wàn)榮一對(duì)情侶失蹤碉输,失蹤者是張志新(化名)和其女友劉穎籽前,沒(méi)想到半個(gè)月后,有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體敷钾,經(jīng)...
    沈念sama閱讀 45,334評(píng)論 1 310
  • 正文 獨(dú)居荒郊野嶺守林人離奇死亡枝哄,尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 37,550評(píng)論 2 333
  • 正文 我和宋清朗相戀三年,在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了阻荒。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片挠锥。...
    茶點(diǎn)故事閱讀 39,727評(píng)論 1 348
  • 序言:一個(gè)原本活蹦亂跳的男人離奇死亡,死狀恐怖侨赡,靈堂內(nèi)的尸體忽然破棺而出蓖租,到底是詐尸還是另有隱情,我是刑警寧澤羊壹,帶...
    沈念sama閱讀 35,428評(píng)論 5 343
  • 正文 年R本政府宣布蓖宦,位于F島的核電站,受9級(jí)特大地震影響油猫,放射性物質(zhì)發(fā)生泄漏稠茂。R本人自食惡果不足惜,卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 41,022評(píng)論 3 326
  • 文/蒙蒙 一情妖、第九天 我趴在偏房一處隱蔽的房頂上張望睬关。 院中可真熱鬧,春花似錦毡证、人聲如沸电爹。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 31,672評(píng)論 0 22
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)丐箩。三九已至,卻和暖如春秦效,著一層夾襖步出監(jiān)牢的瞬間雏蛮,已是汗流浹背涎嚼。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 32,826評(píng)論 1 269
  • 我被黑心中介騙來(lái)泰國(guó)打工阱州, 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留,地道東北人法梯。 一個(gè)月前我還...
    沈念sama閱讀 47,734評(píng)論 2 368
  • 正文 我出身青樓苔货,卻偏偏與公主長(zhǎng)得像犀概,于是被迫代替她去往敵國(guó)和親。 傳聞我的和親對(duì)象是個(gè)殘疾皇子夜惭,可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 44,619評(píng)論 2 354

推薦閱讀更多精彩內(nèi)容