Apache Lucene - Index File Formats(索引文件格式)

Introduction(引言)

This document defines the index file formats used in this version of Lucene. If you are using a different version of Lucene, please consult the copy of docs/ that was distributed with the version you are using.

本文檔定義了V7.0.0版本Lucene中使用的索引文件格式。如果您使用的是不同版本的Lucene良蒸，請查閱docs/隨您使用的版本一起發(fā)布的副本存筏。

This document attempts to provide a high-level definition of the Apache Lucene file formats.

本文檔嘗試提供對Apache Lucene文件格式的高級定義。

Definitions(定義)

The fundamental concepts in Lucene are index, document, field and term.

Lucene的基本概念包括索引(index)，文檔(document)宗雇，域(field)和詞(term)柠偶。

An index contains a sequence of documents.

索引包含一系列文檔。

• A document is a sequence of fields.

• 文檔(document)是一系列域(field)码秉。

• A field is a named sequence of terms.

• 域(field)是一系列經(jīng)過命名的詞(term)。

• A term is a sequence of bytes.

• 詞(term)是一系列字節(jié)(byte)鸡号。

The same sequence of bytes in two different fields is considered a different term. Thus terms are represented as a pair: the string naming the field, and the bytes within the field.

兩個不同域(field)中的相同字節(jié)(byte)序列被認為是不同的詞(term)转砖。因此，詞由一對(要素)表示：域(field)名鲸伴、域(field)內(nèi)字節(jié)組(bytes)府蔗。

Inverted Indexing(倒排索引)

The index stores statistics about terms in order to make term-based search more efficient. Lucene's index falls into the family of indexes known as an inverted index. This is because it can list, for a term, the documents that contain it. This is the inverse of the natural relationship, in which documents list terms.

(倒排索引即)存儲與詞(term)相關(guān)的統(tǒng)計信息的索引(index)，其目的是提高基于詞(term based)的搜索效率挑围。Lucene的索引(index)屬于倒排索引的索引族礁竞。(叫做倒排索引)是因為對于一個詞(term)，此索引(index)可以存儲包含此詞(term)的文檔列表(document list)杉辙。這與某個文件(document)存儲一些詞列表(term list)的自然順序相反模捂。

Types of Fields(域(field)的類型)

In Lucene, fields may be stored, in which case their text is stored in the index literally, in a non-inverted manner. Fields that are inverted are called indexed. A field may be both stored and indexed.

在Lucene中，域(field)可以被存儲(store)蜘矢，此時狂男，域(field)內(nèi)的文本(text)會依照原樣，以非倒轉(zhuǎn)(non-inverted)的方式存入索引中品腹。被倒轉(zhuǎn)(inverted)的域(field)稱作被索引的(indexed)岖食。一個域可以同時被存儲(store)和索引(index)。

The text of a field may be tokenized into terms to be indexed, or the text of a field may be used literally as a term to be indexed. Most fields are tokenized, but sometimes it is useful for certain identifier fields to be indexed literally.

在域(field)中的文本被索引(index)時舞吭，文本既可以被分詞(tokenize)為詞(term)泡垃，也可以(不經(jīng)過分詞)依原樣(literally)作為詞(term)被索引(index)。大多數(shù)域(field)都是經(jīng)過分詞(tokenize)的羡鸥，但有時對于特定標識符域(field)蔑穴，按照字面意思(literally)來索引(index)很有效。

See the Field java docs for more information on Fields.

有關(guān)域的更多信息惧浴，請查閱java docs文檔存和。

Segments(段)

Lucene indexes may be composed of multiple sub-indexes, or segments. Each segment is a fully independent index, which could be searched separately. Indexes evolve by:

Lucene的索引(index)可能有多個子索引或段(segment)構(gòu)成。每個段(segment)是一個完全獨立的索引(index)，可以單獨搜索捐腿。索引(index)演變?yōu)椋?/p>

1. Creating new segments for newly added documents.
2. Merging existing segments.

1. 為新添加的文檔(document)創(chuàng)建新段(segment)纵朋。
2. 合并已有段(segment)。

Searches may involve multiple segments and/or multiple indexes, each index potentially composed of a set of segments.

搜索(search)可能涉及多個段(segment)和/或多個索引(index)茄袖，每個索引(index)可能由一系列段(segment)構(gòu)成操软。

Document Numbers(文檔編號)

Internally, Lucene refers to documents by an integer document number. The first document added to an index is numbered zero, and each subsequent document added gets a number one greater than the previous.

在內(nèi)部，Lucene通過整型(integer)文檔(document number)編號引用文檔(index)绞佩。添加到索引(index)中的首個文檔(document)編號為0寺鸥，后續(xù)添加的文檔(document)其編號依次增長1。

Note that a document's number may change, so caution should be taken when storing these numbers outside of Lucene. In particular, numbers may change in the following situations:

注意品山，文檔(document)的編號可能會改變，因此在Lucene外部存儲文檔編號(document number)時應(yīng)謹慎烤低。尤其是在以下情況中肘交，編號可能會改變。

• The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's base document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the external value is in, and the segment's base value is subtracted. For example two five document segments might be combined, so that the first segment has a base value of zero, and the second of five. Document three from the second segment would have an external value of eight.

• 每段(segment)中存儲的編號僅在本段(segment)內(nèi)唯一扑馁，在更大的上下文(context)中使用前必須進行轉(zhuǎn)換涯呻。標準技術(shù)依據(jù)每段(segment)使用的數(shù)值范圍為其分配一個范圍的值。要將文檔編號(document number)從段(segment)內(nèi)轉(zhuǎn)換為一個外部值腻要，需要給(這個編號)添加段(segment)內(nèi)基本文檔編號(document number)复罐。要將外部值轉(zhuǎn)換為特定段(segment-specific)的值，此段(segment)需檢查外部值是否在數(shù)值范圍內(nèi)雄家，并(使外部值)減去段(segment)的基本文檔編號效诅。例如，兩個五篇文檔(document)的段(segment)將要合并(merge)趟济，那么第一個段(segment)的基本值為0乱投，第二個的基本值為5(之前的文檔里值的總數(shù)，第一段合并前文檔中總數(shù)為0顷编，第二段合并前戚炫，第一段已合并完，文檔總數(shù)為5)媳纬。第二段(segment)的3號文檔(document)的外部值將為8(3+5)双肤。

• When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.

• 當文檔(delete)被刪除(delete)，編號中會產(chǎn)生空白钮惠。隨著索引(index)的通過合并(merge)的演變茅糜，這些(空白)最終會被移除(remove)。被刪除的文檔在段(segment)合并(merge)時會被物理刪除(drop)萌腿。新合并(merge)的段(segment)編號中就沒有空白了限匣。

Index Structure Overview(索引結(jié)構(gòu)概述)

Each segment index maintains the following:

每段(segment)索引(index)都包含以下內(nèi)容：

• Segment info. This contains metadata about a segment, such as the number of documents, what files it uses.

• 段元數(shù)據(jù)：包含段(segment)的元數(shù)據(jù)，例如文檔數(shù)量，使用哪些文件米死。

• Field names. This contains the set of field names used in the index.

• 域名：包含索引(index)中使用的一些列域(field)名锌历。

• Stored Field values. This contains, for each document, a list of attribute-value pairs, where the attributes are field names. These are used to store auxiliary information about the document, such as its title, url, or an identifier to access a database. The set of stored fields are what is returned for each hit when searching. This is keyed by document number.

• 域存儲值：對于每篇文檔(document)，其包含屬性-值對的列表峦筒，其中屬性是域(field)名究西。這被用來存儲文檔(document)的輔助數(shù)據(jù)，如標題物喷，url卤材，訪問數(shù)據(jù)庫的標識符。這一系列域(field)就是搜索(search)時每次命(hit)后要返回的內(nèi)容峦失。以文檔(document)編號作為關(guān)鍵字扇丛。

• Term dictionary. A dictionary containing all of the terms used in all of the indexed fields of all of the documents. The dictionary also contains the number of documents which contain the term, and pointers to the term's frequency and proximity data.

• 詞典：包含所有文檔(document)中所有被索引(indexed)域(field)使用的所有詞(term)的字典(dictionary)。該字典(dictionary)還包含了含有該詞(term)的文檔(document)數(shù)量尉辑，以及指向詞頻(term frequency)帆精、位置信息(proximity data)的指針。

• Term Frequency data. For each term in the dictionary, the numbers of all the documents that contain that term, and the frequency of the term in that document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)

• 詞頻數(shù)據(jù)：對于字典(dictionary)中每個詞(term)隧魄，存儲包含此詞(term)的文檔(document)數(shù)量(df)卓练，及其在每篇文檔(document)的詞頻(frequency)(tf)。條件是IndexOptions.DOCS_ONLY不為omitted购啄。

• Term Proximity data. For each term in the dictionary, the positions that the term occurs in each document. Note that this will not exist if all fields in all documents omit position data.

• 詞位置數(shù)據(jù)：對于字典(dictionary)中每個詞(term)襟企，存儲其在每篇文檔(document)中的出現(xiàn)的位置(position)。注意狮含，如果每篇文檔(document)的每個域(field)都忽略(omit)位置(position)信息顽悼，則此節(jié)點不生成。

• Normalization factors. For each field in each document, a value is stored that is multiplied into the score for hits on that field.

• 標準化因子：對于文檔(document)中的每個域(field)辉川，存儲一個值表蝙，此值將會與域上(field)命中(hit)的得分(score)相乘。

• Term Vectors. For each field in each document, the term vector (sometimes called document vector) may be stored. A term vector consists of term text and term frequency. To add Term Vectors to your index see the Field constructors乓旗。

• 詞向量：對于文檔(document)中的每個域(field)府蛇，可以存儲詞向量(term vector)(有時稱作文檔向量(document vector))。詞向量(term vector)由詞(term)的文本和詞頻(term frequency)組成屿愚。向索引(index)中添加詞向量(term vector)汇跨，請參閱Field的構(gòu)造器。

• Per-document values. Like stored values, these are also keyed by document number, but are generally intended to be loaded into main memory for fast access. Whereas stored values are generally intended for summary results from searches, per-document values are useful for things like scoring factors.

• 單個文檔值：與存儲值相似妆距，單個文檔(document)值也以文檔編號作為關(guān)鍵字穷遂，但其通常加載到主內(nèi)存中以便快速訪問。相比與存儲值通常用作搜索結(jié)果的摘要娱据，單個文檔值對于打分元素(scoring factors)這類過程非常有用蚪黑。

• Live documents. An optional file indicating which documents are live.

• 活躍文檔：一個可選文件，找出哪些文檔(document)在活躍狀態(tài)。

• Point values. Optional pair of files, recording dimensionally indexed fields, to enable fast numeric range filtering and large numeric values like BigInteger and BigDecimal (1D) and geographic shape intersection (2D, 3D).

• 關(guān)鍵點值：一個可選文件忌穿，記錄維度級的被索引域抒寂，用于啟用快速數(shù)值范圍過濾器和BigInteger、BigDecimal(1D)掠剑、地理形狀交叉(geographic shape intersection) (2D, 3D)這樣的大數(shù)值屈芜。

Details on each of these are provided in their linked pages.

每處鏈接的詳細信息都在鏈接頁面中提供。

File Naming(文件命名)

All files belonging to a segment have the same name with varying extensions. The extensions correspond to the different file formats described below. When using the Compound File format (default for small segments) these files (except for the Segment info file, the Lock file, and Deleted documents file) are collapsed into a single .cfs file (see below for details)

同段(segment)中所有文件擁有相同命名與不同擴展名朴译。擴展名對應(yīng)下方對不同文件格式的描述井佑。使用復(fù)合文件格式(默認受眾為小的段(segment))時，這些文件(除段元數(shù)據(jù)文件眠寿、鎖文件和刪除文檔文件)被折疊為單個.cfs文件()躬翁。

Typically, all segments in an index are stored in a single directory, although this is not required.

通常，一個索引(index)下的所有段(segment)存儲在單個目錄(dierctory)中盯拱，盡管這不是必須的姆另。

File names are never re-used. That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is segments_1, then segments_2, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.

文件名不會復(fù)用。也就是說坟乾，當任何文件被保存到目錄時，它會被賦予一個從未使用過的文件名蝶防。這是由簡單的層代(generation)方法實現(xiàn)的甚侣。例如，第一個段(segment)的文件為segments_1间学，然后是segments_2殷费，以此類推。層代(generation)是一系列基于數(shù)字(base 36)-字母表的順序長整數(shù)低葫。

Summary of File Extensions(文件擴展名摘要)

The following table summarizes the names and extensions of the files in Lucene:

下方表格總結(jié)了Lucene中的文件名和擴展名：

Lock File(鎖文件)

The write lock, which is stored in the index directory by default, is named "write.lock". If the lock directory is different from the index directory then the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix derived from the full path to the index directory. When this file is present, a writer is currently modifying the index (adding or removing documents). This lock file ensures that only one writer is modifying the index at a time.

默認存儲在索引(index)目錄(directory)中的寫入鎖(lock)，叫做"write.lock"善涨。如果鎖目錄域與索引(index)目錄不同窒盐，則寫入鎖被命名為"XXXX-write.lock"，"XXXX"是從索引(index)目錄的完整路徑派生的唯一前綴(prefix)钢拧。當文件存在時蟹漓，一個寫入程序(writer)正在修改索引(index)(添加或刪除文檔(document))。此鎖文件確保一次只有一個寫程序(writer)在修改索引(index)源内。

History(歷史)

Compatibility notes are provided in this document, describing how file formats have changed from prior versions:

本文檔提供了兼容性說明葡粒，描述了以前版本中的文件格式是如何變化的：

• In version 2.1, the file format was changed to allow lock-less commits (ie, no more commit lock). The change is fully backwards compatible: you can open a pre-2.1 index for searching or adding/deleting of docs. When the new segments file is saved (committed), it will be written in the new file format (meaning no specific "upgrade" process is needed). But note that once a commit has occurred, pre-2.1 Lucene will not be able to read the index.

• 在V2.1中，文件格式被調(diào)成為允許無鎖提交(即，不再提交鎖)嗽交。這些變更完全向后兼容：您可以打開2.1之前的索引搜索或添加/刪除文檔卿嘲。當新的文檔段文件被保存(提交)時，它將被寫入新的文件格式(意味著不需要特定的"升級"過程)轮纫。但請注意腔寡，一旦發(fā)生提交，2.1之前的Lucene將無法讀取索引掌唾。

• In version 2.3, the file format was changed to allow segments to share a single set of doc store (vectors & stored fields) files. This allows for faster indexing in certain cases. The change is fully backwards compatible (in the same way as the lock-less commits change in 2.1).

• 在V2.3放前，文檔格式被調(diào)整為允許段共享一組文檔存儲(向量和存儲域)文件。這在某些情況下允許更快的索引糯彬。這種變化是完全向后兼容的(與2.1中的無鎖提交相同)凭语。

• In version 2.4, Strings are now written as true UTF-8 byte sequence, not Java's modified UTF-8. See LUCENE-510 for details.

• 在V2.4中，現(xiàn)在字符串以真正的UTF-8字節(jié)序列寫入撩扒，而不是Java的經(jīng)過修改的UTF-8似扔。詳情見LUCENE-510 。

• In version 2.9, an optional opaque Map<String,String> CommitUserData may be passed to IndexWriter's commit methods (and later retrieved), which is recorded in the segments_N file. See LUCENE-1382 for details. Also, diagnostics were added to each segment written recording details about why it was written (due to flush, merge; which OS/JRE was used; etc.). See issue LUCENE-1654 for details.

• 在V2.9中搓谆，一個可選的不透明Map<String炒辉，String>CommituserData可以被傳遞給IndexWriter提交方法(以后被提取)，它被記錄在segments_N文件中泉手。細節(jié)請參見問題LUCENE-1654 黔寇。

• In version 3.0, compressed fields are no longer written to the index (they can still be read, but on merge the new segment will write them, uncompressed). See issue LUCENE-1960 for details.

• 在版本3.0中，壓縮字段不再寫入索引（它們?nèi)匀豢梢员蛔x取斩萌，但在合并時缝裤，新的段會寫入它們，未壓縮）颊郎。詳情請參閱LUCENE-1960 憋飞。

• In version 3.1, segments records the code version that created them. See LUCENE-2720 for details. Additionally segments track explicitly whether or not they have term vectors. See LUCENE-2811 for details.

• 在版本3.1中，段記錄了創(chuàng)建它們的代碼版本姆吭。細節(jié)見 LUCENE-2720榛做。此外段明確跟蹤是否有術(shù)語向量。 細節(jié)見LUCENE-2811猾编。

• In version 3.2, numeric fields are written as natively to stored fields file, previously they were stored in text format only.

• 在版本3.2中瘤睹，數(shù)字字段原生寫入存儲的字段文件，以前它們僅以文本格式存儲答倡。

• In version 3.4, fields can omit position data while still indexing term frequencies.
  在版本3.4中轰传，字段可以省略位置數(shù)據(jù)，同時仍然對術(shù)語頻率編制索引瘪撇。

• In version 4.0, the format of the inverted index became extensible via the Codec api. Fast per-document storage (DocValues) was introduced. Normalization factors need no longer be a single byte, they can be any NumericDocValues. Terms need not be unicode strings, they can be any byte sequence. Term offsets can optionally be indexed into the postings lists. Payloads can be stored in the term vectors.

• 在版本4.0中获茬，倒排索引的格式通過Codecapi 變得可擴展港庄。快速的每文檔存儲（DocValues）被引入恕曲。規(guī)范化因素不再需要一個字節(jié)鹏氧，它們可以是任何一個字節(jié)NumericDocValues。術(shù)語不必是unicode字符串佩谣，它們可以是任何字節(jié)序列把还。術(shù)語偏移量可以選擇性地編入發(fā)布列表中。有效載荷可以存儲在術(shù)語向量中茸俭。

• In version 4.1, the format of the postings list changed to use either of FOR compression or variable-byte encoding, depending upon the frequency of the term. Terms appearing only once were changed to inline directly into the term dictionary. Stored fields are compressed by default.
  在版本4.1中吊履，發(fā)布列表的格式更改為使用FOR壓縮或可變字節(jié)編碼，具體取決于術(shù)語的頻率调鬓。僅出現(xiàn)一次的術(shù)語被改為直接內(nèi)聯(lián)到術(shù)語詞典中艇炎。存儲字段默認為壓縮字段。

• In version 4.2, term vectors are compressed by default. DocValues has a new multi-valued type (SortedSet), that can be used for faceting/grouping/joining on multi-valued fields.

• 在版本4.2中腾窝，術(shù)語向量是默認壓縮的缀踪。DocValues有一個新的多值類型（SortedSet），可用于在多值字段上進行分組/合并虹脯。

• In version 4.5, DocValues were extended to explicitly represent missing values.

• 在版本4.5中驴娃，DocValues被擴展為顯式表示缺失值。

• In version 4.6, FieldInfos were extended to support per-field DocValues generation, to allow updating NumericDocValues fields.

• 在4.6版本中循集，F(xiàn)ieldInfos已擴展為支持每字段DocValues生成托慨，以允許更新NumericDocValues字段。

• In version 4.8, checksum footers were added to the end of each index file for improved data integrity. Specifically, the last 8 bytes of every index file contain the zlib-crc32 checksum of the file.

• 在版本4.8中暇榴，校驗和頁腳被添加到每個索引文件的末尾以提高數(shù)據(jù)完整性。特別是蕉世，每個索引文件的最后8個字節(jié)都包含文件的zlib-crc32校驗和蔼紧。

• In version 4.9, DocValues has a new multi-valued numeric type (SortedNumeric) that is suitable for faceting/sorting/analytics.

• 在版本4.9中，DocValues具有適用于分面/排序/分析的新多值數(shù)值類型（SortedNumeric）狠轻。

• In version 5.4, DocValues have been improved to store more information on disk: addresses for binary fields and ord indexes for multi-valued fields.

• 在版本5.4中奸例，DocValues已得到改進，可在磁盤上存儲更多信息：二進制字段的地址和多值字段的ord索引向楼。

• In version 6.0, Points were added, for multi-dimensional range/distance search.

• 在版本6.0中查吊，添加了點，用于多維范圍/距離搜索湖蜕。

• In version 6.2, new Segment info format that reads/writes the index sort, to support index sorting.

• 在6.2版本中逻卖，讀取/寫入索引排序的新段信息格式支持索引排序。

• In version 7.0, DocValues have been improved to better support sparse doc values thanks to an iterator API.

• 在版本7.0中昭抒，由于使用了迭代器API评也，DocValues得到了改進以更好地支持稀疏文檔值炼杖。

Limitations(限制)

Lucene uses a Java int to refer to document numbers, and the index file format uses an Int32 on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either UInt64 values, or better yet, VInt values which have no limit.

Lucene使用Java int來引用文檔編號，索引文件格式使用Int32 磁盤上的文檔編號盗迟。這是索引文件格式和當前實現(xiàn)的限制坤邪。最終，這些應(yīng)該被替換為任何UInt64值罚缕，或者更好的VInt是沒有限制的值艇纺。