google翻譯一下：
設(shè)置在表引擎中執(zhí)行后臺操作的線程數(shù)（例如么抗，在MergeTree引擎表中的合并）。此設(shè)置是從ClickHouse服務(wù)器啟動時的默認(rèn)配置文件中應(yīng)用的厅翔，無法在用戶會話中進(jìn)行更改乖坠。通過調(diào)整此設(shè)置，可以管理CPU和磁盤負(fù)載刀闷。較小的池使用較少的CPU和磁盤資源熊泵，但是后臺進(jìn)程的執(zhí)行速度較慢，這最終可能會影響查詢性能甸昏。
更改它之前顽分，還請查看相關(guān)的MergeTree設(shè)置，例如number_of_free_entries_in_pool_to_lower_max_size_of_merge和number_of_free_entries_in_pool_to_execute_mutation施蜜。
從描述上看卒蘸，這個值是針對所有表的一個限制。實(shí)際工程實(shí)踐中這個值的控制粒度是什么樣子的翻默？我們PS -T 查看線上一個ch服務(wù)：
161 AsyncBlockInput
1 AsyncMetrics
32 BackgrProcPool
16 BgDistSchPool
1 BgDistSchPool/D
16 BgSchPool
1 BgSchPool/D
9 clickhouse-serv
1 CMD
2 ConfigReloader
2 ExterLdrReload
4 HTTPHandler
86 QueryPipelineEx
1 SessionCleaner
1 SystemLogFlush
14 TCPHandler
發(fā)現(xiàn)這個叫BackgrProcPool的線程數(shù)和線上設(shè)置值匹配缸沃，于是代碼入手看看這個參數(shù)是如何控制后臺merge線程數(shù)的。
首先修械，代碼中找到可能設(shè)置"BackgrProcPool"名字的類趾牧。
BackgroundProcessingPool(int size_,
const PoolSettings & pool_settings = {},
const char * log_name = "BackgroundProcessingPool",
const char * thread_name_ = "BackgrProcPool");
BackgroundProcessingPool::BackgroundProcessingPool(int size_,
const PoolSettings & pool_settings,
const char * log_name,
const char * thread_name_)
: size(size_)
, thread_name(thread_name_)
, settings(pool_settings)
{
logger = &Logger::get(log_name);
LOG_INFO(logger, "Create " << log_name << " with " << size << " threads");
threads.resize(size);
for (auto & thread : threads)
thread = ThreadFromGlobalPool([this] { threadFunction(); });
}
對于每個表，原來clickhouse都會通過一個global的線程池來調(diào)度這些任務(wù)：
//github.com/ClickHouse/src/Storages/StorageMergeTree.h
class StorageMergeTree : public ext::shared_ptr_helper<StorageMergeTree>, public MergeTreeData {
...
private:
/// Task handler for merges, mutations and moves.
BackgroundProcessingPool::TaskHandle merging_mutating_task_handle;
BackgroundProcessingPool::TaskHandle moving_task_handle;
std::vector<MergeTreeData::AlterDataPartTransactionPtr> prepareAlterTransactions(
const ColumnsDescription & new_columns, const IndicesDescription & new_indices, const Context & context);
void loadMutations();
...
}
//github.com/ClickHouse/src/Storages/StorageMergeTree.cpp
void StorageMergeTree::startup()
{
clearOldPartsFromFilesystem();
/// Temporary directories contain incomplete results of merges (after forced restart)
/// and don't allow to reinitialize them, so delete each of them immediately
clearOldTemporaryDirectories(0);
/// NOTE background task will also do the above cleanups periodically.
time_after_previous_cleanup.restart();
if (!getSettings()->disable_background_merges)
merging_mutating_task_handle = global_context.getBackgroundPool().addTask([this] { return mergeMutateTask(); });
if (areBackgroundMovesNeeded())
moving_task_handle = global_context.getBackgroundMovePool().addTask([this] { return movePartsTask(); });
}
//github.com/ClickHouse/src/Interpreters/Context.cpp
BackgroundProcessingPool & Context::getBackgroundPool()
{
auto lock = getLock();
if (!shared->background_pool)
shared->background_pool.emplace(settings.background_pool_size);
return *shared->background_pool;
}
//github.com/ClickHouse/src/Core/Settings.h
M(SettingUInt64, background_pool_size, 16, "Number of threads performing background work for tables (for example, merging in merge tree). Only has meaning at server startup.", 0) \
上述代碼分析基于20.3.15版本肯污，下面的分析基于代碼20.12.5版本翘单。
20.12.5版本優(yōu)化了后臺任務(wù)子模塊的實(shí)現(xiàn)：
https://github.com/ClickHouse/ClickHouse/pull/15983
clickhouse啟動的時候，會load所有的表蹦渣，把表按引擎級別在后臺啟動merge服務(wù)哄芜。
由上序列圖分析，可以看到柬唯，ch后臺merge job的調(diào)度认臊，依賴于一個global的pool context，這個pool的大小锄奢，是tables級別的美尸。
至于我們的老朋友Too many parts 這個錯誤，和我們的線程設(shè)置大小雖然有點(diǎn)關(guān)系斟薇，但更多時候并不是直接原因师坎，這里ch的架構(gòu)師alexy也給出了解釋：
https://github.com/ClickHouse/ClickHouse/issues/11193
寫在最后：
ch大量應(yīng)用了c++的新特性，一個不小心就掉到c++的語法糖中一發(fā)而不可收拾堪滨，越來越像python的寫法看似隨心所欲胯陋，其背后是ch團(tuán)隊(duì)對于操作系統(tǒng)、機(jī)器特性的深入理解。
