1
前言
開發(fā)目的：提高百萬級(jí)數(shù)據(jù)插入效率。
采取方案：利用ThreadPoolTaskExecutor多線程批量插入噪服。
采用技術(shù)：springboot2.1.1+mybatisPlus3.0.6+swagger2.5.0+Lombok1.18.4+postgresql+ThreadPoolTaskExecutor等。
2
具體實(shí)現(xiàn)細(xì)節(jié)
application-dev.properties添加線程池配置信息
# 異步線程配置# 配置核心線程數(shù)
async.executor.thread.core_pool_size = 30
# 配置最大線程數(shù)
async.executor.thread.max_pool_size = 30
# 配置隊(duì)列大小
async.executor.thread.queue_capacity = 99988
# 配置線程池中的線程的名稱前綴
async.executor.thread.name.prefix = async-importDB-
spring容器注入線程池bean對(duì)象
@Configuration
@EnableAsync
@Slf4j
public class ExecutorConfig {
@Value("${async.executor.thread.core_pool_size}")
private int corePoolSize;
@Value("${async.executor.thread.max_pool_size}")
private int maxPoolSize;
@Value("${async.executor.thread.queue_capacity}")
private int queueCapacity;
@Value("${async.executor.thread.name.prefix}")
private String namePrefix;
@Bean(name = "asyncServiceExecutor")
public Executor asyncServiceExecutor() {
log.warn("start asyncServiceExecutor");
//在這里修改
ThreadPoolTaskExecutor executor = new VisiableThreadPoolTaskExecutor();
//配置核心線程數(shù)
executor.setCorePoolSize(corePoolSize);
//配置最大線程數(shù)
executor.setMaxPoolSize(maxPoolSize);
//配置隊(duì)列大小
executor.setQueueCapacity(queueCapacity);
//配置線程池中的線程的名稱前綴
executor.setThreadNamePrefix(namePrefix);
// rejection-policy：當(dāng)pool已經(jīng)達(dá)到max size的時(shí)候仇味，如何處理新任務(wù)
// CALLER_RUNS：不在新線程中執(zhí)行任務(wù)雹顺，而是有調(diào)用者所在的線程來執(zhí)行
executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
//執(zhí)行初始化
executor.initialize();
return executor;
}
}
創(chuàng)建異步線程 業(yè)務(wù)類
@Service
@Slf4j
public class AsyncServiceImpl implements AsyncService {
@Override
@Async("asyncServiceExecutor")
public void executeAsync(List<LogOutputResult> logOutputResults, LogOutputResultMapper logOutputResultMapper, CountDownLatch countDownLatch) {
try{
log.warn("start executeAsync");
//異步線程要做的事情
logOutputResultMapper.addLogOutputResultBatch(logOutputResults);
log.warn("end executeAsync");
}finally {
countDownLatch.countDown();// 很關(guān)鍵, 無論上面程序是否異常必須執(zhí)行countDown,否則await無法釋放
}
}
}
創(chuàng)建多線程批量插入具體業(yè)務(wù)方法
@Override
public int testMultiThread() {
List<LogOutputResult> logOutputResults = getTestData();
//測(cè)試每100條數(shù)據(jù)插入開一個(gè)線程
List<List<LogOutputResult>> lists = ConvertHandler.splitList(logOutputResults, 100);
CountDownLatch countDownLatch = new CountDownLatch(lists.size());
for (List<LogOutputResult> listSub:lists) {
asyncService.executeAsync(listSub, logOutputResultMapper,countDownLatch);
}
try {
countDownLatch.await(); //保證之前的所有的線程都執(zhí)行完成，才會(huì)走下面的嬉愧；
// 這樣就可以在下面拿到所有線程執(zhí)行完的集合結(jié)果
} catch (Exception e) {
log.error("阻塞異常:"+e.getMessage());
}
return logOutputResults.size();
}
模擬2000000 條數(shù)據(jù)進(jìn)行測(cè)試
image.png
總結(jié)
通過以上測(cè)試案列，同樣是導(dǎo)入2000000 條數(shù)據(jù)没酣，多線程耗時(shí)1.67分鐘，單線程耗時(shí)5.75分鐘裕便。通過對(duì)不同線程數(shù)的測(cè)試，發(fā)現(xiàn)不是線程數(shù)越多越好偿衰，具體多少合適挂疆，網(wǎng)上有一個(gè)不成文的算法：CPU核心數(shù)量*2 +2 個(gè)線程下翎。
