1. 創(chuàng)建視頻引擎VideoEngine
函數(shù):VideoEngine::Create()
構(gòu)建VideoEngineImpl對象陨囊,該類繼承了ViEBaseImpl威酒、ViECodecImpl篷帅、ViECaptureImpl、
ViEFileImpl枢贿、ViEImageProcessImpl、ViENetworkImpl抬吟、ViERenderImpl萨咕、
ViERTP_RTCPImpl、ViEExternalCodecImpl火本、VideoEngine類危队,
同時(shí)ViEBaseImpl實(shí)例化視頻共享數(shù)據(jù)單元ViEShareData,將該共享數(shù)據(jù)對象分發(fā)給各Impl類钙畔。
graph TD
A[VideoEngine::Create] --> |vie_impl.cc|B(new VideoEngineImpl:new Config, true)
B --> B1(ViENetworkImpl)
B --> B2(ViECodecImpl)
B --> B3(ViECaptureImpl)
B --> B4(ViEFileImpl)
B --> B5(ViEImageProcessImpl)
B --> B6(ViEBaseImpl)
B --> B7(ViERenderImpl)
B --> B8(ViERTP_RTCPImpl)
B --> B9(ViEExternalCodecImpl)
B --> B10(VideoEngine)
B6--> |vie_shared_data.cc| B6A(ViESharedData::ViESharedData)
B6A --> |創(chuàng)建日志跟蹤對象|B6A1(Trace::CreateTrace)
B6A --> |獲取CPU核心數(shù) number_cores_|B6A2(CpuInfo::DetectNumberOfCores)
B6A --> |創(chuàng)建channel管理對象 channel_manager_|B6A3(new ViEChannelManager)
B6A --> |創(chuàng)建視頻輸入設(shè)備管理對象 input_manager_|B6A4(new ViEInputManager)
B6A --> |創(chuàng)建視頻渲染管理對象 render_manager_| B6A5(new ViERenderManager)
B6A --> |創(chuàng)建并啟動模塊運(yùn)行線程 module_process_thread_: ProcessThread| B6A6(ProcessThread::Create)
其中Impl類都為引擎API接口類茫陆,
用戶通過各API類GetInterface()獲取到的實(shí)際都是該引擎對象對API父類的轉(zhuǎn)換,比如:
ViEBase* ViEBase::GetInterface(VideoEngine* video_engine) {
if (!video_engine) {
return NULL;
}
VideoEngineImpl* vie_impl = static_cast<VideoEngineImpl*>(video_engine);
ViEBaseImpl* vie_base_impl = vie_impl;
(*vie_base_impl)++; // Increase ref count.
return vie_base_impl;
}
2. 創(chuàng)建視頻通道
函數(shù):int ViEBaseImpl::CreateChannel(int& video_channel)
視頻通道創(chuàng)建涉及到各個(gè)視頻相關(guān)模塊創(chuàng)建及觀察者注冊等擎析。
graph TD
A[ViEBaseImpl::CreateChannel] --> |vie_channel_manager.cc| B(ViEChannelManager::CreateChannel)
B --> |vie_channel_group.cc 創(chuàng)建通道組對象,管理remb,bitrateControler等| C(new ChannelGroup)
C --> D(ChannelGroup::CreateSendChannel)
D --> |創(chuàng)建視頻編碼管理對象| E(new ViEEncoder)
E --> |初始化ViEEncoder| F(vie_encoder->Init)
F --> |創(chuàng)建視頻通道| G(new ViEChannel)
G --> |初始化視頻通道| H(channel->Init)
2.1 通道組ChannelGroup
ChannelGroup內(nèi)包含了以下模塊的創(chuàng)建:
1. VieRemb
Remb包發(fā)送模塊:負(fù)責(zé)將RemoteBitrateEstimator模塊的碼率預(yù)測值通過RTCP模塊反饋給遠(yuǎn)端發(fā)送方以便調(diào)整遠(yuǎn)端的發(fā)送碼率
2. BitrateAllocator
碼率變化分配模塊:作為BitrateController與ViEEncoder之間的橋梁簿盅,當(dāng)BitrateController模塊碼率變化時(shí)挥下,通過ViEEncoder在該模塊注冊的觀察者對象將變化的碼率值告訴ViEEncoder。具體函數(shù)為OnNetworkChanged()
ChannelGroup::ChannelGroup(ProcessThread* process_thread, const Config* config)
--> bitrate_allocator_(new BitrateAllocator())
調(diào)用流程
graph TD
A[BitrateControllerImpl::MaybeTriggerOnNetworkChanged] --> |observer_對象為ChannelGroup| B(observer_->OnNetworkChanged)
B --> |vie_channel_group.cc| C(ChannelGroup::OnNetworkChanged)
C --> D(bitrate_allocator_->OnNetworkChanged)
D --> |bitrate_allocator.cc| E(BitrateAllocator::OnNetworkChanged)
E --> |vie_encoder.cc| F(BitrateObserver::OnNetworkChanged)
F --> G(ViEEncoder::OnNetworkChanged)
BitrateAllocator中的BitrateObserver為ViEEncoder在執(zhí)行ViEEncoder::SetEncoder()時(shí)調(diào)用bitrate_allocator_->AddBitrateObserver(bitrate_observer_.get(),...)時(shí)添加進(jìn)去桨醋。
3. BitrateController
發(fā)送端碼率控制模塊:負(fù)責(zé)發(fā)送端碼率控制棚瘟,綜合遠(yuǎn)端Remb反饋的接收碼率與發(fā)送端根據(jù)丟包率等評估的碼率調(diào)整最終發(fā)送碼率。
該模塊繼承Module類喜最,會被注冊到ProcessThread線程內(nèi)運(yùn)行偎蘸,默認(rèn)輪詢時(shí)間為25ms
ChannelGroup::ChannelGroup(ProcessThread* process_thread, const Config* config)
// this即為ChannelGroup,該類繼承了BitrateObserver類,
--> bitrate_controller_(
BitrateController::CreateBitrateController(GetRealTimeClock(), this))
// observer即為傳入的this
--> new BitrateControllerImpl(clock, observer)
--> observer_(observer)
// 注冊模塊到ProcessThread線程內(nèi)
--> process_thread->RegisterModule(bitrate_controller_.get())
故BitrateController內(nèi)observer_對象為ChannelGroup所繼承的BitrateObserver瞬内,
在碼率預(yù)測變化時(shí)會用到迷雪。
4. EncoderStateFeedback
編碼器狀態(tài)反饋模塊:反饋關(guān)鍵幀請求給ViEEncoder
// 構(gòu)建
ChannelGroup::ChannelGroup(ProcessThread* process_thread, const Config* config)
--> encoder_state_feedback_(new EncoderStateFeedback())
// 添加視頻編碼器
ChannelGroup::CreateSendChannel()
--> encoder_state_feedback_->AddEncoder(ssrc, encoder)
// 將EncoderStateFeedback內(nèi)部RtcpIntraFrameObserver觀察對象給
// 到ViEChannel,依次傳給RtpRtcp模塊
ChannelGroup::CreateChannel()
--> new ViEChannel(..., encoder_state_feedback_->GetRtcpIntraFrameObserver(), ...)
--> intra_frame_observer_(intra_frame_observer)
--> ViEChannel::CreateRtpRtcpConfiguration()
--> configuration.intra_frame_callback = intra_frame_observer_;
//將RtcpIntraFrameObserver傳給RtpRtcp模塊
--> rtp_rtcp_.reset(RtpRtcp::CreateRtpRtcp(configuration));
--> ModuleRtpRtcpImpl::ModuleRtpRtcpImpl(const Configuration& configuration)
// 將RtcpIntraFrameObserver傳給RtcpReceiver模塊
--> rtcp_receiver_(configuration.intra_frame_callback)
--> RTCPReceiver::RTCPReceiver
--> _cbRtcpIntraFrameObserver(rtcp_intra_frame_observer)
因此虫蝶,ViEChannel和RTCPReceiver模塊內(nèi)都會有RtcpIntraFrameObserver對象
ViEChannel 內(nèi)由解碼線程DecodingThread調(diào)用章咧,更新解碼速度。目前該功能沒實(shí)現(xiàn)
graph TD
A[ViEChannel::ChannelDecodeProcess] --> |vie_channel.cc| B(intra_frame_observer_->SetVideoDecodingSpeed)
B --> |encoder_state_feedback.cc| C(EncoderStateFeedbackObserver::SetVideoDecodingSpeed)
本地SSRC變化
graph TD
A[RTCPReceiver::SetSsrcs] --> |rtcp_receiver.cc| B(_cbRtcpIntraFrameObserver->OnLocalSsrcChanged)
B --> |encoder_state_feedback.cc| C(EncoderStateFeedbackObserver::OnLocalSsrcChanged)
C --> D(EncoderStateFeedback::OnLocalSsrcChanged)
D --> E(encoder->OnLocalSsrcChanged)
E --> |vie_encoder.cc| F(ViEEncoder::OnLocalSsrcChanged)
視頻中的RTCP包
第一類:關(guān)鍵幀請求
主要包括SLI/PLI/FIR能真,作用是在關(guān)鍵幀丟失無法解碼時(shí)赁严,請求發(fā)送方重新生成并發(fā)送一個(gè)關(guān)鍵幀。
這本質(zhì)是一種重傳粉铐,但是跟傳輸層的重傳的區(qū)別是误澳,它重傳是最新生成的幀。
PLI 是Picture Loss Indication秦躯,SLI 是Slice Loss Indication忆谓。
發(fā)送方接收到接收方反饋的PLI或SLI需要重新讓編碼器生成關(guān)鍵幀并發(fā)送給接收端。FIR 是Full Intra Request踱承,這里面Intra的含義可能很多人不知道倡缠。
Intra的含義是圖像內(nèi)編碼,不需要其他圖像信息即可解碼茎活;Inter指圖像間編碼昙沦,解碼需要參考幀。
故Intra Frame其實(shí)就是指I幀载荔,Inter Frame指P幀或B幀盾饮。那么為什么在PLI和SLI之外還需要一個(gè)FIR呢?
原因是使用場景不同懒熙,F(xiàn)IR更多是在一個(gè)中心化的Video Conference中丘损,新的參與者加入,就需要發(fā)送一個(gè)FIR工扎,其他的參與者給他發(fā)送一個(gè)關(guān)鍵幀這樣才能解碼徘钥,
而PLI和SLI的含義更多是在發(fā)生丟包或解碼錯(cuò)誤時(shí)使用。第二類:重傳請求
主要包括RTX/NACK/RPSI這個(gè)重傳跟關(guān)鍵幀請求的區(qū)別是它可以要求任意幀進(jìn)行重傳
第三類:碼率控制
主要包括REMB/TMMBR/TMMBNTMMBR是Temporal Max Media Bitrate Request肢娘,表示臨時(shí)最大碼率請求呈础。表明接收端當(dāng)前帶寬受限舆驶,告訴發(fā)送端控制碼率。
REMB是ReceiverEstimated Max Bitrate而钞,接收端估計(jì)的最大碼率沙廉。
TMMBN是Temporal Max Media Bitrate Notification
graph TD
A[RTCPReceiver::TriggerCallbacksFromRTCPPacket] --> |收到關(guān)鍵幀請求| A1(_cbRtcpIntraFrameObserver->OnReceivedIntraFrameRequest)
A1 --> |encoder_state_feedback.cc| A1A(EncoderStateFeedbackObserver::OnReceivedIntraFrameRequest)
A1A --> A1B(EncoderStateFeedback::OnReceivedIntraFrameRequest)
A1B --> |vie_encoder.cc| A1C(ViEEncoder::OnReceivedIntraFrameRequest)
A1C --> |video_coding_impl.cc| A1D(VideoCodingModuleImpl::IntraFrameRequest)
A1D --> |video_sender.cc| A1E(VideoSender::IntraFrameRequest)
A1E --> |generic_encoder.cc| A1F(VCMGenericEncoder::RequestFrame)
A1F --> A1G(H264EncoderImpl::Encode)
A --> |收到SLI幀請求| A2(_cbRtcpIntraFrameObserver->OnReceivedSLI)
A2 --> |encoder_state_feedback.cc| A2A(EncoderStateFeedbackObserver::OnReceivedSLI)
A2A --> A2B(EncoderStateFeedback::OnReceivedSLI)
A2B --> |vie_encoder.cc| A2C(ViEEncoder::OnReceivedSLI)
A --> |收到RPSI幀請求| A3(_cbRtcpIntraFrameObserver->OnReceivedRPSI)
A3 --> A3A(EncoderStateFeedbackObserver::OnReceivedRPSI)
A3A --> A3B(EncoderStateFeedback::OnReceivedRPSI)
A3B --> |vie_encoder.cc| A3C(ViEEncoder::OnReceivedRPSI)
5. RemoteBitrateEstimator
接收端碼率預(yù)測模塊:使用卡爾曼算法等進(jìn)行碼率預(yù)測。
該模塊繼承Module類臼节,會被注冊到ProcessThread線程內(nèi)運(yùn)行蓝仲,默認(rèn)輪詢時(shí)間為2000ms
6. CallStats
RTT值反饋模塊:將RTT值反饋到視頻JitterBuffer及RemoteBitrateEstimator模塊做相關(guān)調(diào)整。
該模塊繼承Module類官疲,會被注冊到ProcessThread線程內(nèi)運(yùn)行,默認(rèn)輪詢時(shí)間為1000ms亮隙,
即每1000ms會反饋一次RTT值給JitterBuffer和RemoteBitrateEstimator模塊途凫。
ChannelGroup::ChannelGroup(ProcessThread* process_thread, const Config* config)
--> call_stats_(new CallStats())
// 添加RemoteBitrateEstimator為觀察者
--> call_stats_->RegisterStatsObserver(remote_bitrate_estimator_.get());
//注冊到ProcessThread線程內(nèi)運(yùn)行
--> process_thread->RegisterModule(call_stats_.get());
傳遞到 RtpRtcp 模塊
ChannelGroup::CreateChannel()
--> new ViEChannel(..., call_stats_->rtcp_rtt_stats(), ...)
--> rtt_stats_(rtt_stats) // channel內(nèi)rtt_stats賦值為CallStats內(nèi) RtcpRttStats 對象
--> ViEChannel::CreateRtpRtcpConfiguration()
--> configuration.rtt_stats = rtt_stats_
--> rtp_rtcp_.reset(RtpRtcp::CreateRtpRtcp(configuration)) //該 rtt_stats_ 傳給 RtpRtcp 模塊
--> ModuleRtpRtcpImpl::ModuleRtpRtcpImpl(const Configuration& configuration)
--> rtt_stats_(configuration.rtt_stats) // RtpRtcp 模塊會擁有 CallStats 模塊的 RtcpRttStats 對象
--> call_stats_->RegisterStatsObserver(channel->GetStatsObserver()) // 注冊channel內(nèi)CallStatsObserver為觀察者
最終,CallStats模塊會有兩個(gè)觀察者溢吻,分別為 RemoteBitrateEstimator 和 ViEChannel 內(nèi) CallStatsObserver维费。
調(diào)用流程,首先由RtpRtcp模塊Process()輪詢執(zhí)行上報(bào)RTT更新給CallStats促王,后由CallStats模塊在Process()輪詢時(shí)將更新的RTT分發(fā)給兩個(gè)觀察者犀盟。
graph TD
A[ModuleRtpRtcpImpl::Process] --> |rtp_rtcp_impl.cc| B(rtt_stats_->OnRttUpdate)
B --> |call_stats.cc| C(RtcpObserver::OnRttUpdate)
C --> D(CallStats::OnRttUpdate)
D --> E(CallStats::Process)
E --> |vie_channel.cc| E1(ChannelStatsObserver::OnRttUpdate)
E1 --> E1A(ViEChannel::OnRttUpdate)
E1A --> |video_coding_impl.cc| E1B(VideoCodingModuleImpl::SetReceiveChannelParameters)
E1B --> |video_receiver.cc| E1C(VideoReceiver::SetReceiveChannelParameters)
E1C --> |receiver.cc| E1D(VCMReceiver::UpdateRtt)
E1D --> |jitter_buffer.cc| E1E(VCMJitterBuffer::UpdateRtt)
E --> |vie_channel_group.cc| E2(WrappingBitrateEstimator::OnRttUpdate)
E2 --> |remote_bitrate_estimator_single_stream.cc| E2A(RemoteBitrateEstimatorImpl::OnRttUpdate)
E2A --> |aimd_rate_control.cc| E2B(AimdRateControl::SetRtt)
2.2 ViEEncoder視頻編碼模塊創(chuàng)建
ViEEncoder 負(fù)責(zé)創(chuàng)建維護(hù)VCM、VPM蝇狼、PacedSender模塊阅畴。
VCM(VideoCodingModule), 視頻編碼器相關(guān)管理模塊迅耘。
VPM(VideoProcessingModule), 視頻前處理模塊贱枣。PacedSender,負(fù)責(zé)視頻包有規(guī)律的發(fā)送颤专,不至于因視頻幀太大纽哥,拆分的報(bào)文數(shù)量多導(dǎo)致網(wǎng)絡(luò)擁堵。
==注:PacedSender作為一個(gè)Module栖秕,并沒有注冊到ViEShareData啟動的
ModuleProcessThread處理隊(duì)列春塌,而是在ViEEncoder里面獨(dú)自創(chuàng)建
PacedProcesssThread線程處理==
ChannelGroup::CreateSendChannel()
// 創(chuàng)建ViEEncoder對象
-->new ViEEncoder(..., process_thread_, bitrate_allocator_, bitrate_controller_, ...)
--> ViEEncoder::ViEEncoder
// VCM 模塊
--> VideoCodingModule::Create(this)
// VPM 模塊
--> VideoProcessingModule::Create()
// PacedProcesssThread處理線程,專門處理視頻有序發(fā)送模塊
--> pacer_thread_(ProcessThread::Create())
--> bitrate_observer_.reset(new ViEBitrateObserver(this))
// 將BitrateController對象保存到ViEEncoder簇捍,主要用來在更新
// BitrateController的最大最小及起始碼率
--> bitrate_controller_(bitrate_controller)
// this為ViEEncoder對象
--> pacing_callback_.reset(new ViEPacedSenderCallback(this))
--> paced_sender_.reset(new PacedSender(..., pacing_callback_ , ...))
1 VCM(VideoCodingModule)
2 VPM(VideoProcessingModule)
3 PacedSender
PacedSender模塊會被線程PacedProcessThread定時(shí)執(zhí)行只壳,需要發(fā)送視頻包時(shí)通過回調(diào)pacing_callback_將發(fā)包流程轉(zhuǎn)到ViEEncoder內(nèi)執(zhí)行。
ViEEncoder::ViEEncoder()
--> pacer_thread_(ProcessThread::Create("PacerProcessThread"))
--> pacing_callback_.reset(new ViEPacedSenderCallback(this));
--> paced_sender_.reset(new PacedSender(
Clock::GetRealTimeClock(),
pacing_callback_.get(),
kDefaultStartBitrateKbps,
PacedSender::kDefaultPaceMultiplier * kDefaultStartBitrateKbps,
0));
--> PacedSender::PacedSender(..., callback, ...)
--> callback_(callback)
graph TD
A[PacedSender::Process] --> |paced_sender.cc| A1(PacedSender::SendPacket)
A1 --> A1A(callback_->TimeToSendPacket)
A1A --> |vie_encoder.cc| A1B(ViEPacedSenderCallback::TimeToSendPacket)
A1B --> A1C(ViEEncoder::TimeToSendPacket)
A1C --> A1D(send_payload_router_->TimeToSendPacket)
A1D --> |payload_router.cc| A1E(PayloadRouter::TimeToSendPacket)
A1E --> A1F(rtp_module->TimeToSendPacket)
A1F --> |rtp_rtcp_impl.cc| A1G(ModuleRtpRtcpImpl::TimeToSendPacket)
A1G --> A1H(rtp_sender_.TimeToSendPacket)
A1H --> |rtp_sender.cc| A1I(RTPSender::TimeToSendPacket)
A1I --> A1J(packet_history_.GetPacketAndSetSendTime)
A1J --> A1K(RTPSender::PrepareAndSendPacket)
A1K --> A1L(RTPSender::SendPacketToNetwork)
A1L --> A1M(transport_->SendPacket)
A1M --> |vie_sender.cc| B(ViESender::SendPacket)
A[PacedSender::Process] --> |paced_sender.cc| A2(PacedSender::SendPadding)
A2 --> A2A(callback_->TimeToSendPadding)
A2A --> |vie_encoder.cc| A2B(ViEPacedSenderCallback::TimeToSendPadding)
A2B --> A2C(ViEEncoder::TimeToSendPadding)
A2C --> A2D(send_payload_router_->TimeToSendPadding)
A2D --> |payload_router.cc| A2E(PayloadRouter::TimeToSendPadding)
A2E --> A2F(rtp_module->TimeToSendPadding)
A2F --> |rtp_rtcp_impl.cc| A2G(ModuleRtpRtcpImpl::TimeToSendPadding)
A2G --> A2H(rtp_sender_.TimeToSendPadding)
A2H --> |rtp_sender.cc| A2I(RTPSender::TimeToSendPadding)
A2I --> A2J(RTPSender::TrySendPadData)
A2J --> A2K(RTPSender::SendPadData)
A2K --> A2L(RTPSender::SendPacketToNetwork)
A2L --> A2M(transport_->SendPacket)
A2M --> |vie_sender.cc| B
B --> |udp_transport_impl.cc| C(UdpTransportImpl::SendPacket)