elasticsearch-機(jī)制-文件系統(tǒng)

Directory類(lèi)繼承樹(shù)

在IndicesService的createIndexService函數(shù)中馒铃,IndicesService創(chuàng)建IndexModule實(shí)例對(duì)象痕惋,IndexModule創(chuàng)建IndexStore實(shí)例對(duì)象娃殖,IndexStore創(chuàng)建FsDirectoryService實(shí)例對(duì)象

IndicesService::createIndexService
private synchronized IndexService createIndexService(final String reason, IndexMetaData indexMetaData, IndicesQueryCache indicesQueryCache, IndicesFieldDataCache indicesFieldDataCache, List<IndexEventListener> builtInListeners, IndexingOperationListener... indexingOperationListeners) throws IOException {
    final Index index = indexMetaData.getIndex();
    final Predicate<String> indexNameMatcher = (indexExpression) -> indexNameExpressionResolver.matchesIndex(index.getName(), indexExpression, clusterService.state());
    final IndexSettings idxSettings = new IndexSettings(indexMetaData, this.settings, indexNameMatcher, indexScopeSetting);
    ...
 
    final IndexModule indexModule = new IndexModule(idxSettings, indexStoreConfig, analysisRegistry);  // 創(chuàng)建IndexModule
    ...
 
    return indexModule.newIndexService(nodeEnv, xContentRegistry, this, circuitBreakerService, bigArrays, threadPool, scriptService,
            clusterService, client, indicesQueryCache, mapperRegistry, indicesFieldDataCache);  // 托管IndexModule創(chuàng)建IndexService
}
IndexModule::newIndexService
public IndexService newIndexService(NodeEnvironment environment, NamedXContentRegistry xContentRegistry,
        IndexService.ShardStoreDeleter shardStoreDeleter, CircuitBreakerService circuitBreakerService, BigArrays bigArrays,
        ThreadPool threadPool, ScriptService scriptService,
        ClusterService clusterService, Client client, IndicesQueryCache indicesQueryCache, MapperRegistry mapperRegistry,
        IndicesFieldDataCache indicesFieldDataCache) throws IOException {
 
    ...;
    final String storeType = indexSettings.getValue(INDEX_STORE_TYPE_SETTING);
    final IndexStore store;
    if (Strings.isEmpty(storeType) || isBuiltinType(storeType)) {  // IndexModule提供了5種類(lèi)型的文件系統(tǒng):NIOFS述寡,MMAPFS叶洞,SIMPLEFS,F(xiàn)S螟炫,DEFAULT
        store = new IndexStore(indexSettings, indexStoreConfig);
    } else {  // 也可實(shí)現(xiàn)自定義的IndexStore艺晴,通過(guò)調(diào)用addIndexStore(String type, BiFunction<IndexSettings, IndexStoreConfig, IndexStore> provider)函數(shù)去注冊(cè)自定義IndexStore
        BiFunction<IndexSettings, IndexStoreConfig, IndexStore> factory = storeTypes.get(storeType);
        if (factory == null) {
            throw new IllegalArgumentException("Unknown store type [" + storeType + "]");
        }
        store = factory.apply(indexSettings, indexStoreConfig);
        if (store == null) {
            throw new IllegalStateException("store must not be null");
        }
    }
    indexSettings.getScopedSettings().addSettingsUpdateConsumer(IndexStore.INDEX_STORE_THROTTLE_TYPE_SETTING, store::setType);  // index.store.throttle.type, 節(jié)流控制
    indexSettings.getScopedSettings().addSettingsUpdateConsumer(IndexStore.INDEX_STORE_THROTTLE_MAX_BYTES_PER_SEC_SETTING,
        store::setMaxRate);  // index.store.throttle.max_bytes_per_sec, 節(jié)流控制
    ...;
    return new IndexService(indexSettings, environment, xContentRegistry, new SimilarityService(indexSettings, similarities),
            shardStoreDeleter, analysisRegistry, engineFactory.get(), circuitBreakerService, bigArrays, threadPool, scriptService,
            clusterService, client, queryCache, store, eventListener, searcherWrapperFactory, mapperRegistry,
            indicesFieldDataCache, searchOperationListeners, indexOperationListeners);
}
IndexStore::newDirectoryService
public DirectoryService newDirectoryService(ShardPath path) {
    return new FsDirectoryService(indexSettings, this, path);
}

FsDirectoryService

FsDirectoryService創(chuàng)建目錄時(shí)主要做兩件事:獲取文件鎖封寞,以及根據(jù)指定類(lèi)型創(chuàng)建對(duì)應(yīng)類(lèi)型的目錄對(duì)象

FsDirectoryService::newDirectory
    @Override
    public Directory newDirectory() throws IOException {
        final Path location = path.resolveIndex();  // 獲取目錄位置
        final LockFactory lockFactory = indexSettings.getValue(INDEX_LOCK_FACTOR_SETTING);  // 獲取文件鎖
        Files.createDirectories(location);
        Directory wrapped = newFSDirectory(location, lockFactory);  // 根據(jù)指定類(lèi)型創(chuàng)建對(duì)應(yīng)類(lèi)型的目錄對(duì)象
        Set<String> preLoadExtensions = new HashSet<>(
                indexSettings.getValue(IndexModule.INDEX_STORE_PRE_LOAD_SETTING));
        wrapped = setPreload(wrapped, location, lockFactory, preLoadExtensions);  // MMapFS文件系統(tǒng)在將文件映射到應(yīng)用程序內(nèi)存時(shí)進(jìn)行預(yù)加載操作狈究,將數(shù)據(jù)寫(xiě)進(jìn)MappedByteBuffer
        if (indexSettings.isOnSharedFilesystem()) {
            wrapped = new SleepingLockWrapper(wrapped, 5000);
        }
        return new RateLimitedFSDirectory(wrapped, this, this) ;
    }
  
    protected Directory newFSDirectory(Path location, LockFactory lockFactory) throws IOException {
        final String storeType = indexSettings.getSettings().get(IndexModule.INDEX_STORE_TYPE_SETTING.getKey(),
            IndexModule.Type.FS.getSettingsKey());
        if (IndexModule.Type.FS.match(storeType)) {
            return FSDirectory.open(location, lockFactory); // use lucene defaults
        } else if (IndexModule.Type.DEFAULT.match(storeType)) {
            deprecationLogger.deprecated("[default] store type is deprecated, use [fs] instead");
            return FSDirectory.open(location, lockFactory); // use lucene defaults
        } else if (IndexModule.Type.SIMPLEFS.match(storeType)) {
            return new SimpleFSDirectory(location, lockFactory);
        } else if (IndexModule.Type.NIOFS.match(storeType)) {
            return new NIOFSDirectory(location, lockFactory);
        } else if (IndexModule.Type.MMAPFS.match(storeType)) {
            return new MMapDirectory(location, lockFactory);
        }
        throw new IllegalArgumentException("No directory found for type [" + storeType + "]");
    }
FSDirectoryService文件鎖

由index.store.fs.fs_lock參數(shù)指定抖锥,可選[native, simple]磅废,默認(rèn)native

創(chuàng)建方式 校驗(yàn)方式 解鎖 備注
NativeFSLockFactory Files.createFile(lockFile); // 文件存在時(shí)忽略異常
FileChannel channel = FileChannel.open(realPath, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
FileLock lock = channel.tryLock();
FileTime creationTime = Files.readAttributes(realPath, BasicFileAttributes.class).creationTime();
lock.isValid();
channel.size() == 0;
creationTime.equals(Files.readAttributes(path, BasicFileAttributes.class).creationTime());
沒(méi)有顯示釋放文件鎖【怪海可能是等垃圾回收時(shí)自動(dòng)釋放 依托NIO的文件鎖API實(shí)現(xiàn)
SimpleFSLockFactory Files.createFile(lockFile); // 文件存在時(shí)拋異常
FileTime creationTime = Files.readAttributes(lockFile, BasicFileAttributes.class).creationTime();
creationTime.equals(Files.readAttributes(path, BasicFileAttributes.class).creationTime()); Files.delete(path); 文件創(chuàng)建時(shí)間屬性校驗(yàn)
FSDirectory文件類(lèi)型
type openInput read createOutput and write 備注
FSDirectory Type.FS 或 Type.DEFAULT if (Constants.JRE_IS_64BIT && MMapDirectory.UNMAP_SUPPORTED) {
return new MMapDirectory(path, lockFactory);
} else if (Constants.WINDOWS) {
return new SimpleFSDirectory(path, lockFactory);
} else {
return new NIOFSDirectory(path, lockFactory);
}
OutputStream out = Files.newOutputStream(directory.resolve(name), options);
out.write(b, offset, chunk);
FSDirectory是lucene庫(kù)實(shí)現(xiàn)的文件系統(tǒng)類(lèi)岔帽,根據(jù)操作系統(tǒng)是否為WINDOWS斗遏,操作系統(tǒng)是否為64位鞋邑,操作系統(tǒng)的文件系統(tǒng)是否支持MMAP來(lái)決定使用SimpleFSDirectory账蓉、NIOFSDirectory逾一、MMapDirectory中的一種方式
SimpleFSDirectory Type.SIMPLEFS SeekableByteChannel channel = Files.newByteChannel(path, StandardOpenOption.READ); ByteBuffer bb = ByteBuffer.wrap(b, offset, len);
int i = channel.read(bb);
面向WINDOWS平臺(tái)的文件系統(tǒng)
NIOFSDirectory Type.NIOFS FileChannel fc = FileChannel.open(path, StandardOpenOption.READ); ByteBuffer bb = ByteBuffer.wrap(b, offset, len);
int i = channel.read(bb, pos);
面向LINUX平臺(tái)的NIO庫(kù)的文件系統(tǒng)
MMapDirectory Type.MMAPFS FileChannel c = FileChannel.open(path, StandardOpenOption.READ); MappedByteBuffer buffer = fc.map(MapMode.READ_ONLY, offset + bufferStart, bufSize);
if (preload) { buffer.load(); }
面向LINUX平臺(tái)的支持MMap方式的文件系統(tǒng)
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 序言:七十年代末箱玷,一起剝皮案震驚了整個(gè)濱河市陌宿,隨后出現(xiàn)的幾起案子,更是在濱河造成了極大的恐慌舶得,老刑警劉巖爽蝴,帶你破解...
    沈念sama閱讀 217,406評(píng)論 6 503
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件,死亡現(xiàn)場(chǎng)離奇詭異九孩,居然都是意外死亡发框,警方通過(guò)查閱死者的電腦和手機(jī),發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 92,732評(píng)論 3 393
  • 文/潘曉璐 我一進(jìn)店門(mén),熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)个唧,“玉大人,你說(shuō)我怎么就攤上這事犁河∑翘荩” “怎么了?”我有些...
    開(kāi)封第一講書(shū)人閱讀 163,711評(píng)論 0 353
  • 文/不壞的土叔 我叫張陵,是天一觀的道長(zhǎng)辣苏。 經(jīng)常有香客問(wèn)我哄褒,道長(zhǎng)呐赡,這世上最難降的妖魔是什么骏融? 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 58,380評(píng)論 1 293
  • 正文 為了忘掉前任,我火速辦了婚禮怀泊,結(jié)果婚禮上误趴,老公的妹妹穿的比我還像新娘。我一直安慰自己碧囊,他們只是感情好纤怒,可當(dāng)我...
    茶點(diǎn)故事閱讀 67,432評(píng)論 6 392
  • 文/花漫 我一把揭開(kāi)白布泊窘。 她就那樣靜靜地躺著,像睡著了一般烘豹。 火紅的嫁衣襯著肌膚如雪携悯。 梳的紋絲不亂的頭發(fā)上,一...
    開(kāi)封第一講書(shū)人閱讀 51,301評(píng)論 1 301
  • 那天龟劲,我揣著相機(jī)與錄音轴或,去河邊找鬼。 笑死照雁,一個(gè)胖子當(dāng)著我的面吹牛,可吹牛的內(nèi)容都是我干的萍诱。 我是一名探鬼主播,決...
    沈念sama閱讀 40,145評(píng)論 3 418
  • 文/蒼蘭香墨 我猛地睜開(kāi)眼烫扼,長(zhǎng)吁一口氣:“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼!你這毒婦竟也來(lái)了悟狱?” 一聲冷哼從身側(cè)響起,我...
    開(kāi)封第一講書(shū)人閱讀 39,008評(píng)論 0 276
  • 序言:老撾萬(wàn)榮一對(duì)情侶失蹤苹享,失蹤者是張志新(化名)和其女友劉穎得问,沒(méi)想到半個(gè)月后软免,有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體,經(jīng)...
    沈念sama閱讀 45,443評(píng)論 1 314
  • 正文 獨(dú)居荒郊野嶺守林人離奇死亡膏萧,尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 37,649評(píng)論 3 334
  • 正文 我和宋清朗相戀三年漓骚,在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片榛泛。...
    茶點(diǎn)故事閱讀 39,795評(píng)論 1 347
  • 序言:一個(gè)原本活蹦亂跳的男人離奇死亡蝌蹂,死狀恐怖,靈堂內(nèi)的尸體忽然破棺而出曹锨,到底是詐尸還是另有隱情孤个,我是刑警寧澤,帶...
    沈念sama閱讀 35,501評(píng)論 5 345
  • 正文 年R本政府宣布沛简,位于F島的核電站硼身,受9級(jí)特大地震影響,放射性物質(zhì)發(fā)生泄漏佳遂。R本人自食惡果不足惜,卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 41,119評(píng)論 3 328
  • 文/蒙蒙 一撒顿、第九天 我趴在偏房一處隱蔽的房頂上張望丑罪。 院中可真熱鬧,春花似錦、人聲如沸吩屹。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 31,731評(píng)論 0 22
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)煤搜。三九已至免绿,卻和暖如春,著一層夾襖步出監(jiān)牢的瞬間擦盾,已是汗流浹背嘲驾。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 32,865評(píng)論 1 269
  • 我被黑心中介騙來(lái)泰國(guó)打工, 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留迹卢,地道東北人辽故。 一個(gè)月前我還...
    沈念sama閱讀 47,899評(píng)論 2 370
  • 正文 我出身青樓,卻偏偏與公主長(zhǎng)得像腐碱,于是被迫代替她去往敵國(guó)和親誊垢。 傳聞我的和親對(duì)象是個(gè)殘疾皇子,可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 44,724評(píng)論 2 354