02 本地回環(huán)

本文是基于 Android WebRTC完整入門教程 這篇文章的實(shí)踐過(guò)程記錄胯盯，自己新增的內(nèi)容主要體現(xiàn)在代碼的注釋中

這部分還可以參考大神PIASY的WebRTC native源碼解析系列文章：https://blog.piasy.com/2017/08/30/WebRTC-P2P-part1/index.html

介紹WebRTC中最核心的概念PeerConnection?, 給同一手機(jī)中的前后攝像頭建立虛擬的連接, 相互傳輸畫面

PeerConnection

PeerConnection也就是Peer-to-Peer connection(P2P), 就是兩個(gè)"人"的連接. 雙方分別創(chuàng)建PeerConnection對(duì)象, 然后向?qū)Ψ桨l(fā)送自己的網(wǎng)絡(luò)狀況ICE和多媒體編碼格式SDP(因?yàn)檫@時(shí)候連接還沒(méi)建立, 所以發(fā)送內(nèi)容是通過(guò)服務(wù)器完成的). 當(dāng)雙方網(wǎng)絡(luò)和編碼格式協(xié)商好后, 連接就建立好了, 這時(shí)從PeerConnection中能獲取到對(duì)方的MediaStream數(shù)據(jù)流, 也就能播放對(duì)方的音視頻了

ICE

Interactive Connectivity Establishment, 交互式連接建立. 其實(shí)是一個(gè)整合STUN和TURN的框架, 給它提供STUN和TURN服務(wù)器地址, 它會(huì)自動(dòng)選擇優(yōu)先級(jí)高的進(jìn)行NAT穿透

SDP

Session Description Protocol: 會(huì)話描述協(xié)議. 發(fā)送方的叫Offer, 接受方的叫Answer, 除了名字外沒(méi)有區(qū)別. 就是一些文本描述本地的音視頻編碼和網(wǎng)絡(luò)地址等

主要流程

A(local)和B(remote)代表兩個(gè)人, 初始化PeerConnectionFactory并分別創(chuàng)建PeerConnection?, 并向PeerConnection?添加本地媒體流

1. A創(chuàng)建Offer
2. A保存Offer(set local description)
3. A發(fā)送Offer給B
4. B保存Offer(set remote description)
5. B創(chuàng)建Answer
6. B保存Answer(set local description)
7. B發(fā)送Answer給A
8. A保存Answer(set remote description)
9. A發(fā)送ICE Candidates給B
10. B發(fā)送ICE Candidates給A
11. A, B收到對(duì)方的媒體流并播放

流程圖如下

主要流程.png

準(zhǔn)備步驟

主要是初始化PeerConnectionFactory和使用相機(jī)

public class MainActivity extends AppCompatActivity {
    PeerConnectionFactory peerConnectionFactory;
    PeerConnection peerConnectionLocal;
    PeerConnection peerConnectionRemote;
    SurfaceViewRenderer localView;
    SurfaceViewRenderer remoteView;
    // 在webrtc中耸成，MediaStream代表一個(gè)媒體流疼鸟，AudioTrack及塘，VideoTrack代表音頻”軌道”和視頻“軌道”，如同一個(gè)MP4文件可以包含許多音軌和視頻軌一樣，一個(gè)MediaStream中可以包含多個(gè)AudioTrack和VideoTrack
    MediaStream mediaStreamLocal;
    MediaStream mediaStreamRemote;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        verifyStoragePermissions(this);

        // 創(chuàng)建EglBase對(duì)象, WebRTC 把 EGL 的操作封裝在了 EglBase 中，并針對(duì) EGL10 和 EGL14 提供了不同的實(shí)現(xiàn), 而 OpenGL 的繪制操作則封裝在了 EglRenderer 中
        // 獲取EglBase對(duì)象的上下文
        EglBase.Context eglBaseContext = EglBase.create().getEglBaseContext();

        // create PeerConnectionFactory
        // PeerConnectionFactory負(fù)責(zé)創(chuàng)建PeerConnection吮蛹、VideoTrack、AudioTrack等重要對(duì)象
        // 初始化PeerConnectionFactory
        PeerConnectionFactory.initialize(PeerConnectionFactory.
                InitializationOptions.
                builder(this).
                createInitializationOptions();
        // PeerConnection的選項(xiàng)類
        PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();
        // 視頻編碼工廠類拌屏，負(fù)責(zé)創(chuàng)建視頻編碼類
        DefaultVideoEncoderFactory defaultVideoEncoderFactory = new DefaultVideoEncoderFactory(eglBaseContext,
                true,
                true);
        // 視頻解碼工廠類
        DefaultVideoDecoderFactory defaultVideoDecoderFactory = new DefaultVideoDecoderFactory(eglBaseContext);
        // 設(shè)置PeerConnection工廠類潮针，并創(chuàng)建PeerConnection工廠對(duì)象
        peerConnectionFactory = PeerConnectionFactory.
                builder().
                setOptions(options).
                setVideoEncoderFactory(defaultVideoEncoderFactory).
                setVideoDecoderFactory(defaultVideoDecoderFactory).
                createPeerConnectionFactory();
        
        // SurfaceTextureHelper 負(fù)責(zé)創(chuàng)建 SurfaceTexture，接收 SurfaceTexture 數(shù)據(jù)倚喂，相機(jī)線程的管理
        SurfaceTextureHelper localSurfaceTextureHelper = SurfaceTextureHelper.create("localCaptureThread", eglBaseContext);
        // create VideoCapturer
        // 獲取前置攝像頭
        // WebRTC 視頻采集的接口定義為 VideoCapturer每篷，其中定義了初始化、啟停端圈、銷毀等操作焦读，以及接收啟停事件、數(shù)據(jù)的回調(diào)
        VideoCapturer localVideoCapturer = createCameraCapturer(true);
        // peerConnectionFactory通過(guò)localVideoCapturer創(chuàng)建視頻源
        VideoSource localVideoSource = peerConnectionFactory.createVideoSource(localVideoCapturer.isScreencast());
        localVideoCapturer.initialize(localSurfaceTextureHelper, getApplicationContext(), localVideoSource.getCapturerObserver());
        localVideoCapturer.startCapture(480, 640, 30);//480, 640, 30分別是width, height, fps

        //視頻數(shù)據(jù)在 native 層處理完畢后會(huì)拋出到 VideoRenderer.Callbacks#renderFrame 回調(diào)中舱权，在這里也就是 SurfaceViewRenderer#renderFrame矗晃，而 SurfaceViewRenderer 又會(huì)把數(shù)據(jù)交給 EglRenderer 進(jìn)行渲染
        localView = findViewById(R.id.localView);
        localView.setMirror(true);
        localView.init(eglBaseContext, null);

        // create VideoTrack
        // 創(chuàng)建視頻軌，用于網(wǎng)絡(luò)傳輸
        VideoTrack localVideoTrack = peerConnectionFactory.createVideoTrack("100", localVideoSource);
//        // display in localView
//        localVideoTrack.addSink(localView);

        SurfaceTextureHelper remoteSurfaceTextureHelper = SurfaceTextureHelper.create("remoteCaptureThread", eglBaseContext);
        // create VideoCapturer
        // 獲取后置攝像頭
        VideoCapturer remoteVideoCapturer = createCameraCapturer(false);
        VideoSource remoteVideoSource = peerConnectionFactory.createVideoSource(remoteVideoCapturer.isScreencast());
        remoteVideoCapturer.initialize(remoteSurfaceTextureHelper, getApplicationContext(), remoteVideoSource.getCapturerObserver());
        remoteVideoCapturer.startCapture(480, 640, 30);

        remoteView = findViewById(R.id.remoteView);
        remoteView.setMirror(true);
        remoteView.init(eglBaseContext, null);

        // create VideoTrack
        // VideoTrack是webrtc中視頻流最上層的接口宴倍，它內(nèi)部其實(shí)是經(jīng)過(guò)層層封裝
        VideoTrack remoteVideoTrack = peerConnectionFactory.createVideoTrack("100", remoteVideoSource);
//        // display in remoteView
//        remoteVideoTrack.addSink(remoteView);

        // 向媒體流中添加視頻軌
        mediaStreamLocal = peerConnectionFactory.createLocalMediaStream("mediaStreamLocal");
        mediaStreamLocal.addTrack(localVideoTrack);

        mediaStreamRemote = peerConnectionFactory.createLocalMediaStream("mediaStreamRemote");
        mediaStreamRemote.addTrack(remoteVideoTrack);

        call(mediaStreamLocal, mediaStreamRemote);
    }
    ...
}

使用相機(jī)

對(duì)createCameraCapturer()方法略作修改, 傳入boolean參數(shù)就能分別獲取前后攝像頭, MainActivity.java中

// MainActivity.java中
// isFront==true 獲取前置攝像頭, 反之獲取后置攝像頭
private VideoCapturer createCameraCapturer(boolean isFront){
    Camera1Enumerator enumerator = new Camera1Enumerator(false);
    final String[] deviceNames = enumerator.getDeviceNames();

    // First, try to find front facing cammera
    for (String deviceName : deviceNames){
        if (isFront ? enumerator.isFrontFacing(deviceName) : enumerator.isBackFacing(deviceName)){
            VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);

            if (videoCapturer != null){
                return videoCapturer;
            }
        }
    }

    return null;
}

撥打

建立連接的兩人肯定有一個(gè)是撥打方, 另一個(gè)是接受方. 撥打方創(chuàng)建Offer發(fā)給接受方, 接收方收到后回復(fù)Answer喧兄。

// MainActivity.java中
private void call(MediaStream mediaStreamLocal, MediaStream mediaStreamRemote){
    List<PeerConnection.IceServer> iceServers = new ArrayList<>();
    // 創(chuàng)建本地的peerConnection
    peerConnectionLocal = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("localConnection"){
        @Override
        // peerConnectionLocal發(fā)送收集到的iceCandidate
        public void onIceCandidate(IceCandidate iceCandidate) {
            super.onIceCandidate(iceCandidate);
            // 遠(yuǎn)端添加iceCandidate
            peerConnectionRemote.addIceCandidate(iceCandidate);
        }

        @Override
        public void onAddStream(MediaStream mediaStream) {
            super.onAddStream(mediaStream);
            // 遠(yuǎn)端通過(guò)傳輸過(guò)來(lái)的媒體流將視頻軌添加到localView進(jìn)行渲染
            VideoTrack remoteVideoTrack = mediaStream.videoTracks.get(0);
            // “返回”到主線程，更新應(yīng)用 UI
            runOnUiThread(()->{
                remoteVideoTrack.addSink(localView);
            });
        }
    });

    peerConnectionRemote = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("remoteConnection"){
        @Override
        public void onIceCandidate(IceCandidate iceCandidate) {
            super.onIceCandidate(iceCandidate);
            peerConnectionLocal.addIceCandidate(iceCandidate);
        }

        @Override
        public void onAddStream(MediaStream mediaStream) {
            super.onAddStream(mediaStream);
            VideoTrack localVideoTrack = mediaStream.videoTracks.get(0);
            runOnUiThread(()->{
                localVideoTrack.addSink(remoteView);
            });
        }
    });

    // 添加本地的媒體流
    peerConnectionLocal.addStream(mediaStreamLocal);
    // 創(chuàng)建offer
    peerConnectionLocal.createOffer(new SdpAdapter("local offer sdp"){
        @Override
        public void onCreateSuccess(SessionDescription sessionDescription) {
            super.onCreateSuccess(sessionDescription);
            peerConnectionLocal.setLocalDescription(new SdpAdapter("local set local"), sessionDescription);
            peerConnectionRemote.addStream(mediaStreamRemote);
            peerConnectionRemote.setRemoteDescription(new SdpAdapter("remote set remote"), sessionDescription);
            peerConnectionRemote.createAnswer(new SdpAdapter("remote answer sdp"){
                @Override
                public void onCreateSuccess(SessionDescription sessionDescription) {
                    super.onCreateSuccess(sessionDescription);
                    peerConnectionRemote.setLocalDescription(new SdpAdapter("remote set local"), sessionDescription);
                    peerConnectionLocal.setRemoteDescription(new SdpAdapter("local set remote"), sessionDescription);
                }
            }, new MediaConstraints());
        }
    }, new MediaConstraints());
}

添加PeerConnectionAdapter類作為PeerConnection的觀察者

public class PeerConnectionAdapter implements PeerConnection.Observer {

    private String tag;

    public PeerConnectionAdapter(String tag){
        this.tag = "bo" + tag;
    }

    public void log(String str){
        Log.d(this.tag, str);
    }

    @Override
    public void onSignalingChange(PeerConnection.SignalingState signalingState) {
        log("onSignalingChange" + signalingState);
    }

    @Override
    public void onIceConnectionChange(PeerConnection.IceConnectionState iceConnectionState) {
        log("onIceConnectionChange" + iceConnectionState);
    }

    @Override
    public void onIceConnectionReceivingChange(boolean b) {
        log("onIceConnectionReceivingChange" + b);
    }

    @Override
    public void onIceGatheringChange(PeerConnection.IceGatheringState iceGatheringState) {
        log("onIceGatheringChange" + iceGatheringState);
    }

    @Override
    public void onIceCandidate(IceCandidate iceCandidate) {
        log("onIceCandidate" + iceCandidate);
    }

    @Override
    public void onIceCandidatesRemoved(IceCandidate[] iceCandidates) {
        log("onIceCandidatesRemoved" + iceCandidates);
    }

    @Override
    public void onAddStream(MediaStream mediaStream) {
        log("onAddStream" + mediaStream);
    }

    @Override
    public void onRemoveStream(MediaStream mediaStream) {
        log("onRemoveStream" + mediaStream);
    }

    @Override
    public void onDataChannel(DataChannel dataChannel) {
        log("onDataChannel" + dataChannel);
    }

    @Override
    public void onRenegotiationNeeded() {
        log("onRenegotiationNeeded");
    }

    @Override
    public void onAddTrack(RtpReceiver rtpReceiver, MediaStream[] mediaStreams) {
        log("onAddTrack" + mediaStreams);
    }
}

添加SdpAdapter(繼承SdpObserver)作為Sdp的觀察者

public class SdpAdapter implements SdpObserver {

    private String tag;

    public SdpAdapter(String tag){
        this.tag =  "bo" + tag;
    }

    public void log(String str){
        Log.d(tag, str);
    }

    @Override
    public void onCreateSuccess(SessionDescription sessionDescription) {
        log("onCreateSuccess " + sessionDescription);
    }

    @Override
    public void onSetSuccess() {
        log("onSetSuccess ");
    }

    @Override
    public void onCreateFailure(String s) {
        log("onCreateFailure " + s);
    }

    @Override
    public void onSetFailure(String s) {
        log("onSetFailure " + s);
    }
}

注意: 雖然這里沒(méi)有真正使用到網(wǎng)絡(luò), 但是要添加網(wǎng)絡(luò)權(quán)限

<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>

Android6.0及以上申請(qǐng)權(quán)限

private static final int REQUEST_ALL = 1;

private static String[] PERMISSIONS_ALL = {
        "android.permission.CAMERA",
        "android.permission.RECORD_AUDIO",
        "android.permission.INTERNET",
        "android.permission.ACCESS_NETWORK_STATE"
};

//然后通過(guò)一個(gè)函數(shù)來(lái)申請(qǐng)
public static void verifyStoragePermissions(Activity activity) {
    try {
        int permission = 0;
        //檢測(cè)所有需要的權(quán)限
        for(String temp : PERMISSIONS_ALL){
            permission = ActivityCompat.checkSelfPermission(activity, temp);
            if (permission != PackageManager.PERMISSION_GRANTED){
                break;
            }
        }

        if (permission != PackageManager.PERMISSION_GRANTED) {
            // 沒(méi)有寫的權(quán)限啊楚，去申請(qǐng)寫的權(quán)限，會(huì)彈出對(duì)話框
            ActivityCompat.requestPermissions(activity, PERMISSIONS_ALL,REQUEST_ALL);
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

網(wǎng)上大部分本地回環(huán)(Loopback)的Demo都只用到一個(gè)攝像頭, 這里使用到同一個(gè)手機(jī)的前后攝像頭, 把它們當(dāng)做兩個(gè)客戶端, 建立模擬連接, 發(fā)送媒體數(shù)據(jù). 這跟實(shí)際WebRTC工作流程非常接近了, 只有一點(diǎn)差別--這里的數(shù)據(jù)傳輸是內(nèi)存共享, 而實(shí)際是通過(guò)網(wǎng)絡(luò)發(fā)送