基本概念

ApplicationConfiguration 應(yīng)用配置信息

• 定義模塊的ModuleConfiguration
• ModuleConfiguration定義ModuleProvider的ProviderConfiguration

ModuleManager 模塊管理器

• 負(fù)責(zé)管理服務(wù)內(nèi)模塊實(shí)例
• 持有全部loaded的ModuleDefine信息

ModuleDefine 模塊信息

• 模塊的名稱定義
• 關(guān)聯(lián)ModuleProvider提供者
• 定義需要的Service接口類信息Class[]

ModuleProvider 提供者

• 負(fù)責(zé)模塊具體功能的運(yùn)行
• 關(guān)聯(lián)ModuleDefine
• 提供ModuleConfig信息
• 定義依賴的模塊名稱信息String[]
• 通過(guò)prepare注冊(cè)需要的Service接口實(shí)例
• 擁有ModuleManager信息

Service

抽象設(shè)計(jì)的接口類

啟動(dòng)的過(guò)程

從OAPServerStartUp代碼上了解sw的代碼風(fēng)格和設(shè)計(jì)

public class OAPServerStartUp {
    private static final Logger logger = LoggerFactory.getLogger(OAPServerStartUp.class);
    public static void main(String[] args) {
        String mode = System.getProperty("mode");
        RunningMode.setMode(mode);
        ApplicationConfigLoader configLoader = new ApplicationConfigLoader();
        ModuleManager manager = new ModuleManager();
        try {
            // 配置信息的解析权纤，讀取application.yml的配置信息
            ApplicationConfiguration applicationConfiguration = configLoader.load();
            
            // 關(guān)鍵步驟 容器初始化，各模塊的啟動(dòng)
            manager.init(applicationConfiguration);
            
            // 添加啟動(dòng)時(shí)間的監(jiān)控信息
            manager.find(TelemetryModule.NAME).provider().getService(MetricsCreator.class).createGauge("uptime",
                    "oap server start up time", MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE)
                    // Set uptime to second
                    .setValue(System.currentTimeMillis() / 1000d);
            if (RunningMode.isInitMode()) {
                logger.info("OAP starts up in init mode successfully, exit now...");
                System.exit(0);
            }
        } catch (Throwable t) {
            logger.error(t.getMessage(), t);
            System.exit(1);
        }
    }
}

main方法的代碼比較簡(jiǎn)單就是加載配置文件乌妒，調(diào)用manager的init初始化
配置信息以coreModule為例說(shuō)明

# ModuleConfiguration
core: # ModuleName
#PrividerConfiguration
  default: #ProviderName
    # Provider的Properties配置
    role: ${SW_CORE_ROLE:Mixed} 
    restHost: ${SW_CORE_REST_HOST:0.0.0.0}
    restPort: ${SW_CORE_REST_PORT:12800}
    restContextPath: ${SW_CORE_REST_CONTEXT_PATH:/}
    gRPCHost: ${SW_CORE_GRPC_HOST:0.0.0.0}
    gRPCPort: ${SW_CORE_GRPC_PORT:11800}
    downsampling:
      - Hour
      - Day
      - Month
    enableDataKeeperExecutor: ${SW_CORE_ENABLE_DATA_KEEPER_EXECUTOR:true} 
    dataKeeperExecutePeriod: ${SW_CORE_DATA_KEEPER_EXECUTE_PERIOD:5} 
    recordDataTTL: ${SW_CORE_RECORD_DATA_TTL:90} 
    minuteMetricsDataTTL: ${SW_CORE_MINUTE_METRIC_DATA_TTL:90} 
    hourMetricsDataTTL: ${SW_CORE_HOUR_METRIC_DATA_TTL:36} 
    dayMetricsDataTTL: ${SW_CORE_DAY_METRIC_DATA_TTL:45} 
    monthMetricsDataTTL: ${SW_CORE_MONTH_METRIC_DATA_TTL:18} 
    enableDatabaseSession: ${SW_CORE_ENABLE_DATABASE_SESSION:true}

繼續(xù)跟蹤manager.init方法

    public void init(
        ApplicationConfiguration applicationConfiguration) throws ModuleNotFoundException, ProviderNotFoundException, ServiceNotProvidedException, CycleDependencyException, ModuleConfigException, ModuleStartException {
        String[] moduleNames = applicationConfiguration.moduleList();
        // 1. SPI機(jī)制查找所有模塊實(shí)現(xiàn)類 ModuleDefine
        ServiceLoader<ModuleDefine> moduleServiceLoader = ServiceLoader.load(ModuleDefine.class);
        // SPI機(jī)制查實(shí)模塊提供者實(shí)現(xiàn)類 ModuleProvider
        ServiceLoader<ModuleProvider> moduleProviderLoader = ServiceLoader.load(ModuleProvider.class);
        // 配置文件定義的模塊
        LinkedList<String> moduleList = new LinkedList<>(Arrays.asList(moduleNames));
        // 檢查配置文件中定義的模塊汹想，系統(tǒng)是是否存在
        for (ModuleDefine module : moduleServiceLoader) {
            for (String moduleName : moduleNames) {
                if (moduleName.equals(module.name())) {
                    ModuleDefine newInstance;
                    try {
                        // 反射創(chuàng)建實(shí)例
                        newInstance = module.getClass().newInstance();
                    } catch (InstantiationException | IllegalAccessException e) {
                        throw new ModuleNotFoundException(e);
                    }
                    // 2. 模塊準(zhǔn)備流程
                    newInstance.prepare(this, applicationConfiguration.getModuleConfiguration(moduleName), moduleProviderLoader);
                    loadedModules.put(moduleName, newInstance);
                    moduleList.remove(moduleName);
                }
            }
        }
        // Finish prepare stage
        isInPrepareStage = false;
        // 存在不知名module配置則拋異常
        if (moduleList.size() > 0) {
            throw new ModuleNotFoundException(moduleList.toString() + " missing.");
        }
        // 3. 模塊加載完成 啟動(dòng)服務(wù)
        BootstrapFlow bootstrapFlow = new BootstrapFlow(loadedModules);
        bootstrapFlow.start(this);
        bootstrapFlow.notifyAfterCompleted();
    }

通過(guò)分析上面標(biāo)注了信息的步驟來(lái)了解代碼流程

1. SPI機(jī)制加載模塊信息

通過(guò)JavaSPI機(jī)制加載META-INF/services目錄下定義的類和實(shí)例
以CoreModule為例可以看到META-INF/services目錄下定義的
org.apache.skywalking.oap.server.library.module.ModuleDefine
org.apache.skywalking.oap.server.library.module.ModuleProvider
文件
ModuleDefine

org.apache.skywalking.oap.server.core.storage.StorageModule
org.apache.skywalking.oap.server.core.cluster.ClusterModule
org.apache.skywalking.oap.server.core.CoreModule
org.apache.skywalking.oap.server.core.query.QueryModule
org.apache.skywalking.oap.server.core.alarm.AlarmModule
org.apache.skywalking.oap.server.core.exporter.ExporterModule

ModuleProvider

org.apache.skywalking.oap.server.core.CoreModuleProvider

加載完成后進(jìn)行遍歷，通過(guò)反射創(chuàng)建對(duì)象實(shí)例

2. ModuleDefine的prepare方法

    void prepare(ModuleManager moduleManager, ApplicationConfiguration.ModuleConfiguration configuration,
        ServiceLoader<ModuleProvider> moduleProviderLoader) throws ProviderNotFoundException, ServiceNotProvidedException, ModuleConfigException, ModuleStartException {
        // 遍歷provider
        for (ModuleProvider provider : moduleProviderLoader) {
            // 配置中包含provider才處理
            if (!configuration.has(provider.name())) {
                continue;
            }
            // 判斷Provider定義的Module是否 當(dāng)前的ModuleDefine相同撤蚊。建立一對(duì)一的關(guān)系
            // 這里避免多個(gè)provider關(guān)聯(lián)同一個(gè)Module實(shí)例
            if (provider.module().equals(getClass())) {
                if (loadedProvider == null) {
                    loadedProvider = provider;
                    loadedProvider.setManager(moduleManager);
                    loadedProvider.setModuleDefine(this);
                } else {
                    // ModuleName + " module has one " + ProviderName + "[" + ProviderClass + "] provider already, "
                    throw new DuplicateProviderException(this.name() + " module has one " + loadedProvider.name() + "[" + loadedProvider.getClass().getName() + "] provider already, "
                        + provider.name() + "[" + provider.getClass().getName() + "] is defined as 2nd provider.");
                }
            }

        }

        if (loadedProvider == null) {
            throw new ProviderNotFoundException(this.name() + " module no provider exists.");
        }

        logger.info("Prepare the {} provider in {} module.", loadedProvider.name(), this.name());
        try {
            // 加載ModuleConfig配置欧宜，由provider管理
            copyProperties(loadedProvider.createConfigBeanIfAbsent(), configuration.getProviderConfiguration(loadedProvider.name()), this.name(), loadedProvider.name());
        } catch (IllegalAccessException e) {
            throw new ModuleConfigException(this.name() + " module config transport to config bean failure.", e);
        }
        // provider準(zhǔn)備階段
        loadedProvider.prepare();
    }

ModuleDefine的準(zhǔn)備階段主要工作是關(guān)聯(lián)一個(gè)Provider，有且必須只能關(guān)聯(lián)到一個(gè)provider拴魄，否則報(bào)錯(cuò)。建立完關(guān)聯(lián)關(guān)系后加載配置信息到Provider席镀，再執(zhí)行Provider的準(zhǔn)備階段匹中。
繼續(xù)跟蹤Provider的prepare。ModuleProvider的prepare方法是抽象方法豪诲，由具體子類實(shí)現(xiàn)赏殃。以CoreModuleProvider為例

 @Override public void prepare() throws ServiceNotProvidedException, ModuleStartException {
    // 暫時(shí)不展開(kāi)伴鳖，省略代碼
        this.registerServiceImplementation(ConfigService.class, new ConfigService(moduleConfig));
        this.registerServiceImplementation(DownsamplingConfigService.class, new DownsamplingConfigService(moduleConfig.getDownsampling()));
        this.registerServiceImplementation(GRPCHandlerRegister.class, new GRPCHandlerRegisterImpl(grpcServer));
        this.registerServiceImplementation(JettyHandlerRegister.class, new JettyHandlerRegisterImpl(jettyServer));
        this.registerServiceImplementation(IComponentLibraryCatalogService.class, new ComponentLibraryCatalogService());
        this.registerServiceImplementation(SourceReceiver.class, receiver);
        WorkerInstancesService instancesService = new WorkerInstancesService();
        this.registerServiceImplementation(IWorkerInstanceGetter.class, instancesService);
        this.registerServiceImplementation(IWorkerInstanceSetter.class, instancesService);
        this.registerServiceImplementation(RemoteSenderService.class, new RemoteSenderService(getManager()));
        this.registerServiceImplementation(IModelSetter.class, storageModels);
        this.registerServiceImplementation(IModelGetter.class, storageModels);
        this.registerServiceImplementation(IModelOverride.class, storageModels);
        this.registerServiceImplementation(ServiceInventoryCache.class, new ServiceInventoryCache(getManager()));
        this.registerServiceImplementation(IServiceInventoryRegister.class, new ServiceInventoryRegister(getManager()));
        this.registerServiceImplementation(ServiceInstanceInventoryCache.class, new ServiceInstanceInventoryCache(getManager()));
        this.registerServiceImplementation(IServiceInstanceInventoryRegister.class, new ServiceInstanceInventoryRegister(getManager()));
        this.registerServiceImplementation(EndpointInventoryCache.class, new EndpointInventoryCache(getManager()));
        this.registerServiceImplementation(IEndpointInventoryRegister.class, new EndpointInventoryRegister(getManager()));
        this.registerServiceImplementation(NetworkAddressInventoryCache.class, new NetworkAddressInventoryCache(getManager()));
        this.registerServiceImplementation(INetworkAddressInventoryRegister.class, new NetworkAddressInventoryRegister(getManager()));
        this.registerServiceImplementation(TopologyQueryService.class, new TopologyQueryService(getManager()));
        this.registerServiceImplementation(MetricQueryService.class, new MetricQueryService(getManager()));
        this.registerServiceImplementation(TraceQueryService.class, new TraceQueryService(getManager()));
        this.registerServiceImplementation(LogQueryService.class, new LogQueryService(getManager()));
        this.registerServiceImplementation(MetadataQueryService.class, new MetadataQueryService(getManager()));
        this.registerServiceImplementation(AggregationQueryService.class, new AggregationQueryService(getManager()));
        this.registerServiceImplementation(AlarmQueryService.class, new AlarmQueryService(getManager()));
        this.registerServiceImplementation(TopNRecordsQueryService.class, new TopNRecordsQueryService(getManager()));
        this.registerServiceImplementation(CommandService.class, new CommandService(getManager()));
        this.registerServiceImplementation(RemoteClientManager.class, remoteClientManager);
    }

先不展開(kāi)講CoreModuleProvider的具體功能，對(duì)比其他Provider，prepare都執(zhí)行了 this.registerServiceImplementation()方法用以注冊(cè)Provider需要的Service實(shí)例抓艳。
當(dāng)所有ModuleDefine執(zhí)行完prepare,準(zhǔn)備階段結(jié)束。

3. BootstrapFlow的運(yùn)行

BootstrapFlow的創(chuàng)建摔敛，關(guān)鍵看makeSequence方法

    BootstrapFlow(Map<String, ModuleDefine> loadedModules) throws CycleDependencyException {
        this.loadedModules = loadedModules;
        startupSequence = new LinkedList<>();
        makeSequence();
    }

makeSequence方法,實(shí)現(xiàn)的功能是通過(guò)自旋判斷Provider需要的Modules是否已經(jīng)準(zhǔn)備完成蝇裤，當(dāng)出現(xiàn)循環(huán)依賴是拋出異常

    private void makeSequence() throws CycleDependencyException {
        List<ModuleProvider> allProviders = new ArrayList<>();
        // 創(chuàng)建待啟動(dòng)提供者集合
        loadedModules.forEach((moduleName, module) -> allProviders.add(module.provider()));
        // 檢測(cè)提供者模塊依賴模塊啟動(dòng)
        do {
            int numOfToBeSequenced = allProviders.size();
            for (int i = 0; i < allProviders.size(); i++) {
                ModuleProvider provider = allProviders.get(i);
                // 模塊提供者配置了依賴模塊
                String[] requiredModules = provider.requiredModules();
                if (CollectionUtils.isNotEmpty(requiredModules)) {
                    boolean isAllRequiredModuleStarted = true;
                    for (String module : requiredModules) {
                        // find module in all ready existed startupSequence
                        boolean exist = false;
                        for (ModuleProvider moduleProvider : startupSequence) {
                            if (moduleProvider.getModuleName().equals(module)) {
                                exist = true;
                                break;
                            }
                        }
                        // 依賴模塊未啟動(dòng) 跳出循環(huán) 下次再檢測(cè)
                        if (!exist) {
                            isAllRequiredModuleStarted = false;
                            break;
                        }
                    }
                    if (isAllRequiredModuleStarted) {
                        startupSequence.add(provider);
                        // 啟動(dòng)完成移除
                        allProviders.remove(i);
                        i--;
                    }
                } else {
                    // 無(wú)依賴 啟動(dòng)完成
                    startupSequence.add(provider);
                    allProviders.remove(i);
                    i--;
                }
            }
            // 循環(huán)依賴檢測(cè) MA依賴MB,MB依賴MA，則allProviders永遠(yuǎn)無(wú)法remove,出現(xiàn)循環(huán)
            if (numOfToBeSequenced == allProviders.size()) {
                StringBuilder unSequencedProviders = new StringBuilder();
                allProviders.forEach(provider -> unSequencedProviders.append(provider.getModuleName()).append("[provider=").append(provider.getClass().getName()).append("]\n"));
                throw new CycleDependencyException("Exist cycle module dependencies in \n" + unSequencedProviders.substring(0, unSequencedProviders.length() - 1));
            }
        }
        while (allProviders.size() != 0);
    }

再看BootstFlow.start(),這里負(fù)責(zé)的是Provider的啟動(dòng)階段秦士，并做了module和provider中service關(guān)系的校驗(yàn)

    void start(
        ModuleManager moduleManager) throws ModuleNotFoundException, ServiceNotProvidedException, ModuleStartException {
        for (ModuleProvider provider : startupSequence) {
            String[] requiredModules = provider.requiredModules();
            if (requiredModules != null) {
                for (String module : requiredModules) {
                    if (!moduleManager.has(module)) {
                        throw new ModuleNotFoundException(module + " is required by " + provider.getModuleName()
                            + "." + provider.name() + ", but not found.");
                    }
                }
            }
            logger.info("start the provider {} in {} module.", provider.name(), provider.getModuleName());
            // 判斷關(guān)聯(lián)的ModuleDefine定義的services class[] 實(shí)例是否已經(jīng)注冊(cè)進(jìn)來(lái)
            provider.requiredCheck(provider.getModule().services());
            // 啟動(dòng)
            provider.start();
        }
    }

provider的start是抽象方法缺厉，由具體實(shí)現(xiàn)類完成。
BootstrapFlow執(zhí)行完成start后隧土，調(diào)用notifyAfterCompleted提针，遍歷啟動(dòng)完成的provider執(zhí)行notifyAfterCompleted方法，用于啟動(dòng)完成的回調(diào)曹傀。同樣notifyAfterCompleted也是抽象方法辐脖，由具體實(shí)現(xiàn)類實(shí)現(xiàn)。
至此OAP Server啟動(dòng)流程完成皆愉。
可通過(guò)流程圖輔助了解

image.png

文章基于6.4.0版本
有些術(shù)語(yǔ)憑自己理解定義可能不太嚴(yán)謹(jǐn)嗜价，歡迎指正艇抠。