live555中視頻和音頻是分別進行編碼的椿访，如何實現(xiàn)兩者的同步呢?
如果可以做到讓視頻和音頻的時間戳，都與NTP時間保持同步埠通，就可達到音視頻同步的目的赎离。
Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems overpacket-switched, variable-latency data networks.
在live555中是如何實現(xiàn)這種機制的呢?
總體思路是：
- RTSP服務端利用RTCP協(xié)議中的Sender Report將NTP Timestamp發(fā)送到RTSP客戶端逛犹。
-
RTSP客戶端(數(shù)據(jù)的接收方)把A/V的RTP時間戳同步到RTCP的絕對時間(NTP Timestamp)端辱，實現(xiàn)A/V同步。
這個絕對時間就是當前時間距離Jan 1 1900 00:00:00的差值虽画。
首先看一下未加入同步機制的時間戳代碼:
void RTPReceptionStats::noteIncomingPacket(u_int16_t seqNum,
u_int32_t rtpTimestamp,
unsigned timestampFrequency,
Boolean useForJitterCalculation,
struct timeval& resultPresentationTime,
Boolean& resultHasBeenSyncedUsingRTCP,
unsigned packetSize)
{
...
// Record the inter-packet delay
struct timeval timeNow;
gettimeofday(&timeNow, NULL);
...
// Return the 'presentation time' that corresponds to "rtpTimestamp":
if (fSyncTime.tv_sec == 0 && fSyncTime.tv_usec == 0)
{
// This is the first timestamp that we've seen, so use the current
// 'wall clock' time as the synchronization time. (This will be
// corrected later when we receive RTCP SRs.)
fSyncTimestamp = rtpTimestamp; // 首個RTP Timestamp
fSyncTime = timeNow; // 使用當前系統(tǒng)時間作為初始參考時間戳
}
int timestampDiff = rtpTimestamp - fSyncTimestamp;
// Note: This works even if the timestamp wraps around
// (as long as "int" is 32 bits)
// Divide this by the timestamp frequency to get real time:
double timeDiff = timestampDiff/(double)timestampFrequency;
// Add this to the 'sync time' to get our result:
unsigned const million = 1000000;
unsigned seconds, uSeconds;
if (timeDiff >= 0.0)
{
// 計算時間戳
seconds = fSyncTime.tv_sec + (unsigned)(timeDiff);
uSeconds = fSyncTime.tv_usec + (unsigned)((timeDiff - (unsigned)timeDiff)*million);
if (uSeconds >= million)
{
uSeconds -= million;
++seconds;
}
}
else
{
timeDiff = -timeDiff;
seconds = fSyncTime.tv_sec - (unsigned)(timeDiff);
uSeconds = fSyncTime.tv_usec - (unsigned)((timeDiff - (unsigned)timeDiff)*million);
if ((int)uSeconds < 0)
{
uSeconds += million;
--seconds;
}
}
resultPresentationTime.tv_sec = seconds;
resultPresentationTime.tv_usec = uSeconds;
resultHasBeenSyncedUsingRTCP = fHasBeenSynchronized;
// Save these as the new synchronization timestamp & time:
fSyncTimestamp = rtpTimestamp;
fSyncTime = resultPresentationTime;
fPreviousPacketRTPTimestamp = rtpTimestamp;
}
其中有兩個重要的參數(shù): fSyncTimestamp和fSyncTime;
class RTPReceptionStats {
...
private:
// Used to convert from RTP timestamp to 'wall clock' time:
Boolean fHasBeenSynchronized;
u_int32_t fSyncTimestamp;
struct timeval fSyncTime;
};
-
fSyncTimestamp
RTP Timestamp, 默認第N幀的rtpTimestamp為第N+1幀的fSyncTimestamp舞蔽。 -
fSyncTime
'wall clock' time, 默認第N幀的'wall clock' time為第N+1幀的fSyncTime。
RTPReceptionStats::noteIncomingPacket的實質(zhì)是:
將 RTP timestamp 轉(zhuǎn)換為 'wall clock' time码撰。
獲取首個RTP時渗柿，將系統(tǒng)時間作為首個'wall clock' time。
后續(xù)，當RTP timestamp發(fā)生變化時朵栖，要將變化的部分轉(zhuǎn)換為real time:
int timestampDiff = rtpTimestamp - fSyncTimestamp;
// Divide this by the timestamp frequency to get real time:
double timeDiff = timestampDiff/(double)timestampFrequency;
然后將該部分改變反映到'wall clock' time上颊亮， 如:
seconds = fSyncTime.tv_sec + (unsigned)(timeDiff);
uSeconds = fSyncTime.tv_usec + (unsigned)((timeDiff - (unsigned)timeDiff)*million);
可以看出以上的邏輯中，完全取決于系統(tǒng)時間的精確度陨溅，沒有任何校正機制终惑。
live555是在哪里實現(xiàn)時間校正的呢?
答案是利用RTSP客戶端(數(shù)據(jù)的接收者)利用RTCP返回的Sender Report, 然后利用其中的NTP Timestamp和RTP timestamp, 對fSyncTimestamp和fSyncTime進行校正。
Part of Sender Report RTCP Packet
校正程序如下:
void RTPReceptionStats::noteIncomingSR(u_int32_t ntpTimestampMSW,
u_int32_t ntpTimestampLSW,
u_int32_t rtpTimestamp)
{
fLastReceivedSR_NTPmsw = ntpTimestampMSW;
fLastReceivedSR_NTPlsw = ntpTimestampLSW;
gettimeofday(&fLastReceivedSR_time, NULL);
// Use this SR to update time synchronization information:
// ntpTimestampMSW : NTP timestamp, most significant word (64位NTP時間戳的高32位)
fSyncTimestamp = rtpTimestamp;
fSyncTime.tv_sec = ntpTimestampMSW - 0x83AA7E80; // 1/1/1900 -> 1/1/1970
// ntpTimestampLSW : NTP timestamp, least significant word (64位NTP時間戳的低32位)
double microseconds = (ntpTimestampLSW * 15625.0) / 0x04000000; // 10^6/2^32
fSyncTime.tv_usec = (unsigned)(microseconds + 0.5);
}
通過Sender Report门扇，分別對視頻和音頻的時間及時進行校正雹有，即可保證視音頻同步。
References:
https://en.wikipedia.org/wiki/Network_Time_Protocol
RTP: A Transport Protocol for Real-Time Applications
