本文試圖分析UE4的游戲網絡通信模塊,采用從先勾勒出主框架再到深入細節(jié)的探索模式。

概念介紹

UE4 GamePlay的組成:

1. World
   游戲世界,關于游戲的一切都發(fā)生在其中,游戲運行時必有一個World存在冲呢。

2. Actor
   游戲世界中的角色(如:房子辛馆、怪物、英雄)欧瘪。

3. PlayerController
   玩家在游戲世界中的代理。

4. LocalPlayer
   表示本地玩家炕泳，它對應一個客戶端窗口區(qū)域昭殉。

5. NetConnection
   表示一個client-server的網絡連接。

6. NetDriver
   網絡通信對象(非OS的驅動)辕翰，提供一個UDP Socket用于與外界進行網絡通信夺衍，是游戲網絡數據的出口和進口點。管理當前的NetConnections,
   對于Server端的NetDriver對象管理多個NetConnections喜命，對于Client端的NetDriver對象管理一個NetConnection沟沙。

7. Channel
   信道, 每個NetConnection存在多個信道, 按信道的功能分類如下:
   a. ControlChannel
   用于交換控制消息(control message)，例如:連接請求壁榕、斷開請求等矛紫。每個Connection只有一個該類型通道。
   b. VoiceChannel
   用于交換語音聊天數據牌里。每個Connection只有一個該類型通道颊咬。
   c. ActorChannel
   用于同步游戲中角色數據和進行RPC。存在多個該類型通道實例牡辽，每個實例為一個replicated Actor服務喳篇。

   LocalPlayer和NetConnection的概念非常類似于Linux上的本地和遠程終端概念, 如下圖所示:

LinuxTTY_UE4CSMode.jpg

網絡通信中的數據處理和控制

先舉個物流運輸案例, 有兩個工廠M和N,工廠M有產線A、B催享、C杭隙，工廠N有產線X,Y,Z，產線A因妙、B痰憎、C的產品分別供給產線X,Y,Z。現在有一些卡車(載重t)負責M和N之間的運輸工作∨屎現在描述下從M輸送產品到N的流程:

1. A將自己的產品打包成箱铣耘，如果該產品比較大則分裝到多個箱子中并標記每個箱子表示是產品的一個部分(part0,part1…)。這些箱子搬運到卡車上以故。
2. 同理B蜗细、C也是這樣操作。
3. 當卡車裝滿后怒详，則卡車啟動前往N炉媒；如果是緊急情況，則卡車未滿也要出發(fā)昆烁。注意：有可能卡車可能一次運輸不了某個產品吊骤，需要多次運輸。
4. 卡車到達N后静尼，進行卸貨白粉，根據箱子上的產品標記传泊，將箱子給相應的產線X,Y,Z。假如這時候有個A產線產品部分到達鸭巴，那么X需要等下一趟運輸眷细，繼續(xù)取A的部分產品，直至組成完整產品鹃祖，才進行處理溪椎。同理B,C也是這樣操作。

備注：這里我們假定產品運輸途中不會丟失恬口。
UE4 World的網絡數據流程與上面類似池磁。UE4的網絡數據被定義為Bunch（一束，等價于上述的箱子概念）楷兽。

• 數據發(fā)送流程

  
  
  Send_Bunch.jpg

• 數據接收流程

  
  
  Recv_Bunch.jpg

相關源碼:

Engine\Source\Runtime\Engine\Classes\Engine\Channel.h
Engine\Source\Runtime\Engine\Classes\Engine\ControlChannel.h
Engine\Source\Runtime\Engine\Classes\Engine\ActorChannel.h
Engine\Source\Runtime\Engine\Classes\Engine\VoiceChannel.h&.cpp
Engine\Source\Runtime\Engine\Classes\Engine\DataChannel.h &.cpp
Engine\Source\Runtime\Engine\Classes\Engine\NetConnection.h & .cpp
Engine\Source\Runtime\CoreUObject\Public\UObject\CoreNet.h
Engine\Source\Runtime\Online\OnlineSubsystemUtils\Classes\IpConnection.h
Engine\Source\Runtime\PacketHandlers\PacketHandler\Public\PacketHandler.h
Engine\Source\Runtime\PacketHandlers\PacketHandler\Private\PacketHandler.cpp //對即將發(fā)送出的packet和讀入packet進行處理 
Engine\Source\Runtime\Engine\Private\PacketHandlers\StatelessConnectHandlerComponent.cpp
Engine\Source\Runtime\Engine\Public\PacketHandlers\StatelessConnectHandlerComponent.h // PacketHandler component for implementing a stateless (non-memory-consuming)    
connection handshake Partially based on the Datagram Transport Layer Security protocol.

名詞解釋:

1. Bunch 從Channel產出的叫FoutBunch地熄，輸入給Channel的叫FinBuch
2. Packet 從Socket讀入\輸出叫packet.
3. Open 發(fā)出去的packet且尚未收到ack成為Open狀態(tài)。

備注: 本小節(jié)不考慮數據的生成和解析(也就是生產線上是如何制造出產品的和使用產品的),只需要知道要發(fā)送和接收數據被定義為Bunch(一個箱子)芯杀。

網絡數據接收調用堆棧:

1. Actor Channel Stack:

   
   
   Recv_Actor_Channel.jpg

2. Control Channel Stack:

   
   
   Recv_Control_Channel.jpg
3. Voice-Channel Stack:
   TODO:

函數說明:
1.void UNetConnection::ReceivedRawPacket( void* InData, int32 Count )
處理接收到的網絡數據InData, 首先用HandlePacket鏈處理(如handshake操作)端考，如果剩余數據則繼續(xù)算出有效的packet長度，并調用ReceivedPacket(). 其次該函數還做了些網絡包流量統(tǒng)計揭厚。
2.void UNetConnection::ReceivedPacket( FBitReader& Reader );
分析packet中的數據流, 如果是Ack消息則響應ack消息; 如果是Bunch數據却特，則派發(fā)它們; 最后根據需要發(fā)送Ack消息。
3.void UChannel::ReceivedRawBunch( FInBunch & Bunch, bool & bOutSkipAck );
判斷是否bReliable和亂序, 如果亂序則將bunch放入排序隊列中筛圆，等待前面缺失的bunch到達; 否則調用ReceivedNextBunch()裂明。
4.bool UChannel::ReceivedSequencedBunch( FInBunch& Bunch )
處理一個完整的bunch.它會調用virtual void ReceivedBunch( FInBunch& Bunch ) ;函數。
然后繼續(xù)處理被緩存的滿足順序性的bunch.
5.virtual void ReceivedBunch( FInBunch& Bunch );
不同類型的的UChannel會重寫該函數太援，進行相應的處理闽晦。后面會分別分析它們。

網絡數據發(fā)送調用堆棧:

Send_Actor_Channel.jpg

函數說明：
1.FPacketIdRange UChannel::SendBunch( FOutBunch* Bunch, bool Merge );
合并或將較大的bunch拆分為多個小bunch發(fā)送出去提岔。返回這些bunch占據的pakcet的范圍(用PacketId區(qū)間表示)
2.int32 UChannel::SendRawBunch(FOutBunch* OutBunch, bool Merge);

int32 UChannel::SendRawBunch(FOutBunch* OutBunch, bool Merge)
{
    if ( Connection->bResendAllDataSinceOpen )
    {
        check( OpenPacketId.First != INDEX_NONE );
        check( OpenPacketId.Last != INDEX_NONE );
        return Connection->SendRawBunch( *OutBunch, Merge );
    }

    // Send the raw bunch.
    OutBunch->ReceivedAck = 0;
    int32 PacketId = Connection->SendRawBunch(*OutBunch, Merge);
    if( OpenPacketId.First==INDEX_NONE && OpenedLocally )
        OpenPacketId = FPacketIdRange(PacketId);
    if( OutBunch->bClose )
        SetClosingFlag();

    return PacketId;
}

3.int32 UNetConnection::SendRawBunch( FOutBunch& Bunch, bool InAllowMerge );
將bunch信息寫入輸出流中仙蛉，并調用WriteBitsToSendBuffer()
4.int32 UNetConnection::WriteBitsToSendBuffer()
寫入sendBuffer, 可能會調用FlushNet。返回此次的PacketId.
5.void UNetConnection::FlushNet();
調用UIpConnection::LowLevelSend()發(fā)送數據碱蒙。
6.void UIpConnection::LowLevelSend(void* Data, int32 CountBytes, int32 CountBits);
這里會先調用PacketHandler對data進行處理(目前是在data之前加入些信息)荠瘪。請參考void StatelessConnectHandlerComponent::Outgoing(FBitWriter& Packet)。

有效數據包格式(從bit0開始向后排)
1.這里是PacketHander加入的數據
2.Package-Id (18 bits MAX_PACKETID)
3.IsAck (1 bit)
4.If(IsAck==true)
AckPacketId (18 bits)
bHasServerFrameTime (1 bit)
5.否則赛惩，屬于Bunch.

Control bit,
bOpen  bit
bClose bit
bDormant bit
bReliable bit
ChIndex  MAX_CHANNELS
bHasPackageMapExports bit
bHasMustBeMappedGUIDs bit
bPartial bit
bPartialIntial bit
bPartialFinal bit
ChType  CHTYPE_MAX
BunchDataBits  MAX_PACKET*8   // 純凈的bunch數據長度

6.純凈的數據(真正的數據)

下面做一個傳輸過程總結:

傳輸過程小結圖示.jpg

UDP實現可靠性

在上節(jié)中闡述了數據的傳輸流程哀墓，我們沒有關注數據在傳輸過程中丟失的情況。本節(jié)將分析如何做到可靠地傳輸喷兼。
數據的標識:

1. Bunch用 ChIndex和ChSequenceId(reliable時才會用)
2. Packet用PacketId (一直遞增)

對于可靠的bunch包, 發(fā)送端必須要收到Acked, 接收端必須發(fā)送Acked.
相關函數:

• FOutBunch* UChannel::PrepBunch(FOutBunch* Bunch, FOutBunch* OutBunch, bool Merge);
• void UNetConnection::ReceivedPacket( FBitReader& Reader );
• void UNetConnection::ReceivedNak( int32 NakPacketId );
• void UChannel::ReceivedNak( int32 NakPacketId );

這塊代碼要考慮到如下問題：

1. Packet丟失或不按順序到達
2. Bunch丟失或接收到重復的bunch

Q: 在實現可靠性傳輸時,使用了Sequence號, 但是MAX_CHSEQUENCE才是1024篮绰，如果發(fā)生wrap怎么辦?
A: 這里用的技巧是跟據已經收到的SequenceId(int32)進行推算出正在處理的Bunch的SequenceID，比如當前的SequenceId是10994, 收到的Bunch SequenceID為512褒搔，那么該Sequence的絕對ID為10752阶牍，采用就近原則: 10994 % 1024 = 754, 754-512 < (1024/2), 所以這個SequenceId = 10994 - (754-512) = 10752。

  if ( Bunch.bReliable )
  {
        if ( InternalAck )
    {
         // We can derive the sequence for 100% reliable connections
         Bunch.ChSequence = InReliable[Bunch.ChIndex] + 1;
    }
    else
    {
       // If this is a reliable bunch, use the last processed reliable sequence to read the new reliable sequence
      Bunch.ChSequence = MakeRelative( Reader.ReadInt( MAX_CHSEQUENCE ), InReliable[Bunch.ChIndex], MAX_CHSEQUENCE );
        }
  } 

其中MakeReltive()函數從Wrap的ID計算出絕對的SequenceId星瘾。

Bunch數據的生產

上面闡述了Bunch的發(fā)送和接收流程走孽，本節(jié)將追蹤下Bunch的生產。也就是產線上產品的產生過程琳状。

1. UControlChannel數據的產生
   請參考DataChannel.h中的DEFINE_CONTROL_CHANNEL_MESSAGE_XXX宏磕瓷。很簡單，之間的序列化念逞。
2. UActorChannel數據的產生
   bool UActorChannel::ReplicateActor()負責Actor網絡數據的生成困食。

• Actor屬性標記

/** Structure to hold and pass around transient flags used during replication. */
struct FReplicationFlags
{
    union
    {
        struct
        {
            /** True if replicating actor is owned by the player controller on the target machine. */
            uint32 bNetOwner:1;
            /** True if this is the initial network update for the replicating actor. */
            uint32 bNetInitial:1;
            /** True if this is actor is RemoteRole simulated. */
            uint32 bNetSimulated:1;
            /** True if this is actor's ReplicatedMovement.bRepPhysics flag is true. */
            uint32 bRepPhysics:1;
            /** True if this actor is replicating on a replay connection. */
            uint32 bReplay:1;
        };

        uint32  Value;
    };
    FReplicationFlags()
    {
        Value = 0;
    }
};

• ReplicateActor()函數中代碼片段:

    // ----------------------------------------------------------
    // Replicate Actor and Component properties and RPCs
    // ----------------------------------------------------------

#if USE_NETWORK_PROFILER 
    const uint32 ActorReplicateStartTime = GNetworkProfiler.IsTrackingEnabled() ? FPlatformTime::Cycles() : 0;
#endif

    if (!bIsNewlyReplicationPaused)
    {
        // The Actor
        WroteSomethingImportant |= ActorReplicator->ReplicateProperties(Bunch, RepFlags);

        // The SubObjects
        WroteSomethingImportant |= Actor->ReplicateSubobjects(this, &Bunch, &RepFlags);

        if (Connection->bResendAllDataSinceOpen)
        {
            if (WroteSomethingImportant)
            {
                SendBunch(&Bunch, 1);
            }

            MemMark.Pop();

            bIsReplicatingActor = false;

            return WroteSomethingImportant;
        }

        // Look for deleted subobjects
        for (auto RepComp = ReplicationMap.CreateIterator(); RepComp; ++RepComp)
        {
            if (!RepComp.Key().IsValid())
            {
                // Write a deletion content header:
                WriteContentBlockForSubObjectDelete(Bunch, RepComp.Value()->ObjectNetGUID);

                WroteSomethingImportant = true;
                Bunch.bReliable = true;

                RepComp.Value()->CleanUp();
                RepComp.RemoveCurrent();
            }
        }
    }

• 負責數據同步的算法類和數據結構
  源文件: DataReplication.h, DataReplication.cpp, RepLayout.h, RepLayout.cpp
  • FrepChangedPropertyTracker
    Rep-Property的變化跟蹤器, 每個對象有一個。

    This class is used to store the change list for a group of properties of a particular actor/object
    This information is shared across connections when possible.
    

  • FrepLayout
    對象的Rep-Layout(需要網絡rep的 對象的屬性布局), 每個類有一個翎承。
  • FReplicationChangelistMgr
    每個對象一個硕盹，記錄了對象屬性值的變化歷史記錄，就靠它了(非常關鍵叨咖，可以理解為瘩例，它是一個變化采樣器).

    /** 
     * FReplicationChangelistMgr manages a list of change lists for a particular replicated object that have occurred since the object started replicating
     * Once the history is completely full, the very first changelist will then be merged with the next one (freeing a slot)
     *     This way we always have the entire history for join in progress players
     * This information is then used by all connections, to share the compare work needed to determine what to send each connection
     * Connections will send any changelist that is new since the last time the connection checked
     */
    

  • FObjectReplicator
    針對每個連接和每個對象有一個Replicator實例。

    /** FObjectReplicator
     *   Generic class that replicates properties for an object.
     *   All delta/diffing work is done in this class. 
     *  Its primary job is to produce and consume chunks of properties/RPCs:
     *
     *      |----------------|
     *      | NetGUID ObjRef |
     *      |----------------|
     *      |                |     
     *      | Properties...  |
     *      |                | 
     *      | RPCs...        |
     *      |                |
     *      |----------------|
     *      | </End Tag>     |
     *      |----------------|
     * 
     */
    

  • FrepState
    每個對象針對每個連接的Rep歷史數據,放在FobjectReplicator對象中甸各。 FrepLayout負責操作它垛贤，不記錄數據。(參考Quake3的服務器同步設計)趣倾。
  • UPackageMapClient
    維護Object和FNetGUID的映射聘惦。每個Connection擁有一個。
    Maps objects and names to and from indices for network communication.

這塊代碼比較復雜儒恋，這個小節(jié)只做了些關鍵點闡述善绎，細節(jié)還是要閱讀源碼。
疑問:
在Replicate Actor時诫尽，Actor的子對象是如何同步的涂邀，如何解決互相引用問題？

客戶端連接服務器的流程

下面的控制消息用于客戶端和服務器的握手連接箱锐，閱讀源碼時注意查找發(fā)送時機比勉。

// message type definitions
DEFINE_CONTROL_CHANNEL_MESSAGE_TWOPARAM(Hello, 0, uint8, uint32); // initial client connection message
DEFINE_CONTROL_CHANNEL_MESSAGE_THREEPARAM(Welcome, 1, FString, FString, FString); // server tells client they're ok'ed to load the server's level
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(Upgrade, 2, uint32); // server tells client their version is incompatible
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(Challenge, 3, FString); // server sends client challenge string to verify integrity
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(Netspeed, 4, int32); // client sends requested transfer rate
DEFINE_CONTROL_CHANNEL_MESSAGE_THREEPARAM(Login, 5, FString, FString, FUniqueNetIdRepl); // client requests to be admitted to the game
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(Failure, 6, FString); // indicates connection failure
DEFINE_CONTROL_CHANNEL_MESSAGE_ZEROPARAM(Join, 9); // final join request (spawns PlayerController)
DEFINE_CONTROL_CHANNEL_MESSAGE_TWOPARAM(JoinSplit, 10, FString, FUniqueNetIdRepl); // child player (splitscreen) join request
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(Skip, 12, FGuid); // client request to skip an optional package
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(Abort, 13, FGuid); // client informs server that it aborted a not-yet-verified package due to an UNLOAD request
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(PCSwap, 15, int32); // client tells server it has completed a swap of its Connection->Actor
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(ActorChannelFailure, 16, int32); // client tells server that it failed to open an Actor channel sent by the server (e.g. couldn't serialize Actor archetype)
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(DebugText, 17, FString); // debug text sent to all clients or to server
DEFINE_CONTROL_CHANNEL_MESSAGE_TWOPARAM(NetGUIDAssign, 18, FNetworkGUID, FString); // Explicit NetworkGUID assignment. This is rare and only happens if a netguid is only serialized client->server (this msg goes server->client to tell client what ID to use in that case)
DEFINE_CONTROL_CHANNEL_MESSAGE_ONEPARAM(SecurityViolation, 19, FString); // server tells client that it has violated security and has been disconnected
DEFINE_CONTROL_CHANNEL_MESSAGE_TWOPARAM(GameSpecific, 20, uint8, FString); // custom game-specific message routed to UGameInstance for processing

網絡同步的高層策略

本節(jié)闡述在網絡游戲世界同步過程中，決定哪些對象在什么時機同步到客戶端的策略驹止。我們不可能把所有的對象都同步到客戶端（只同步當前跟客戶端玩家相關性的對象）浩聋，也不可能在一幀中同步所有數據，需要根據優(yōu)先級和同步頻率有節(jié)奏地進行臊恋。

相關函數流程(服務器執(zhí)行)：

1. void UNetDriver::TickFlush(float DeltaSeconds);
2. int32 UNetDriver::ServerReplicateActors(float DeltaSeconds);
   同步工作在此函數中執(zhí)行衣洁，關鍵的流程如下：
   • ServerReplicateActors_BuildConsiderList( ConsiderList, ServerTickTime );
     生成需要同步的Actors列表
   • 針對每個Client Connection:
     1. ServerReplicateActors_PrioritizeActors( Connection, ConnectionViewers, ConsiderList, bCPUSaturated, PriorityList, PriorityActors );
        對ConsiderList中的Actors的NetPriority進行評估，然后排序抖仅，結果放入PriorityActors坊夫。
     2. ServerReplicateActors_ProcessPrioritizedActors( Connection, ConnectionViewers, PriorityActors, FinalSortedCount, Updated );
        遍歷PriorityActors砖第，針對每個Actor執(zhí)行replicated(主要工作在Channel->ReplicateActor())。

重點函數

• bool UActorChannel::ReplicateActor();

ReplicateActor.jpg

• bool FObjectReplicator::ReplicateProperties( FOutBunch & Bunch, FReplicationFlags RepFlags );

ReplicateProperties.jpg