This article analyzes the storage details of Elasticsearch rally's tsdb dataset. Through this article, we can analyze the storage capacity and compression of each field of rally's tsdb dataset.
Data Description
Elasticsearch rally tsdb data: https://github.com/elastic/rally-tracks/tree/master/tsdb
image.png
Test Description
Test case
case1: configure time_series
case2: enable ali_codec docvalues compression
case3: enable ali_codec docvalues compression + do not store _source
ali_codec: ali_codec is a codec plugin, deveolp by alibaba, it use to compress lucene data. in the test case, docvalues use zstd to compression column data.
do not store _source: we modify some elasticsearch server code, that can no store _source, _id, and _seq_no in lucene.
Test results
Time series data includes tags data, metrics data, and @timestamp, _tsid fields.
Metadata includes fields such as _id, _source, and _seq_no.
image.png
Field details:
image.png
Test details
case1: Configure time_series
read segment:_3tn, count=122613113
store summery: size=9.8gb, fieldCount=2, fileds=_source,_id
docvalue summery: size=3.4gb, fieldCount=176
points summery: size=2.6gb, fieldCount=96
inverted summery: size=912mb, postings size=604.7mb fieldCount=82
segment totalSize=17.4gb, tsdbSize=5.7gb, percent=32.98%, otherSize=11.6gb, percent=67.02%, summery:
fields size percent desc
_id and _source 10.6gb 61.41% [docvalues: 0b] [points: 0b] [terms: 901mb] [posting: 0b] [summery: 9.8gb]
metrics and tags 4.8gb 28.13% [docvalues: 2.6gb] [points: 1.6gb] [terms: 10.9mb] [posting: 604mb] [summery: 0b]
_seq_no 998.2mb 5.60% [docvalues: 409.2mb] [points: 589mb] [terms: 0b] [posting: 0b] [summery: 0b]
@timestamp 831.2mb 4.66% [docvalues: 409.2mb] [points: 422mb] [terms: 0b] [posting: 0b] [summery: 0b]
_tsid 34.8mb 0.20% [docvalues: 34.8mb] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
_field_names 764kb 0.00% [docvalues: 0b] [points: 0b] [terms: 15b] [posting: 764kb] [summery: 0b]
_version 0b 0.00% [docvalues: 0b] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
_primary_term 0b 0.00% [docvalues: 0b] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
case2: enable ali_codec docvalues compression
read segment:_3uh, count=122613113
store summery: size=7.5gb, fieldCount=2, fileds=_source,_id
docvalue summery: size=1.4gb, fieldCount=176
points summery: size=2.6gb, fieldCount=96
inverted summery: size=911.4mb, postings size=604.7mb fieldCount=82
segment totalSize=13.1gb, tsdbSize=3.7gb, percent=28.31%, otherSize=9.4gb, percent=71.69%, summery:
fields size percent desc
_id and _source 8.4gb 63.79% [docvalues: 0b] [points: 0b] [terms: 900.4mb] [posting: 0b] [summery: 7.5gb]
metrics and tags 3.2gb 24.30% [docvalues: 936.4mb] [points: 1.6gb] [terms: 10.9mb] [posting: 604mb] [summery: 0b]
_seq_no 1gb 7.89% [docvalues: 475.2mb] [points: 589mb] [terms: 0b] [posting: 0b] [summery: 0b]
@timestamp 538.9mb 3.99% [docvalues: 116.9mb] [points: 422mb] [terms: 0b] [posting: 0b] [summery: 0b]
_tsid 1.9mb 0.01% [docvalues: 1.9mb] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
_field_names 764kb 0.01% [docvalues: 0b] [points: 0b] [terms: 15b] [posting: 764kb] [summery: 0b]
_version 8b 0.00% [docvalues: 8b] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
_primary_term 8b 0.00% [docvalues: 8b] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
case3: enable ali_codec docvalues compression + do not store _source
read segment:_3ot, count=122613113
store summery: size=971.6mb, fieldCount=1, fileds=_id
docvalue summery: size=1.4gb, fieldCount=176
points summery: size=2.6gb, fieldCount=96
inverted summery: size=916.5mb, postings size=604.7mb fieldCount=82
segment totalSize=6.6gb, tsdbSize=3.7gb, percent=56.49%, otherSize=2.8gb, percent=43.51%, summery:
fields size percent desc
metrics and tags 3.2gb 48.49% [docvalues: 936.4mb] [points: 1.6gb] [terms: 10.9mb] [posting: 604mb] [summery: 0b]
_id and _source 1.8gb 27.76% [docvalues: 0b] [points: 0b] [terms: 905.6mb] [posting: 0b] [summery: 971.6mb]
_seq_no 1gb 15.73% [docvalues: 474.8mb] [points: 589mb] [terms: 0b] [posting: 0b] [summery: 0b]
@timestamp 538.9mb 7.97% [docvalues: 116.9mb] [points: 422mb] [terms: 0b] [posting: 0b] [summery: 0b]
_tsid 1.9mb 0.03% [docvalues: 1.9mb] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
_field_names 764kb 0.01% [docvalues: 0b] [points: 0b] [terms: 15b] [posting: 764kb] [summery: 0b]
_version 8b 0.00% [docvalues: 8b] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
_primary_term 8b 0.00% [docvalues: 8b] [points: 0b] [terms: 0b] [posting: 0b] [summery: 0b]
