HPC-SGE環(huán)境下的任務(wù)投遞方式
??HPC指的是高可用計(jì)算集群吮廉，是相對(duì)于單機(jī)服務(wù)或節(jié)點(diǎn)而言的苞尝，通過(guò)節(jié)點(diǎn)或集群間的分布式結(jié)構(gòu)解決任務(wù)需求中的高算力、高存儲(chǔ)宦芦、強(qiáng)動(dòng)態(tài)調(diào)節(jié)等功能宙址。
??每個(gè)節(jié)點(diǎn)在集群中是以相對(duì)獨(dú)立的姿態(tài)在系統(tǒng)中存在，在多節(jié)點(diǎn)的集群中调卑，需要任務(wù)投遞時(shí)選擇合適的任務(wù)投遞工具管理任務(wù)和集群抡砂。
??一般使用SGE集群時(shí)，采用的投遞工具為QSUB恬涧，具體的使用方式參見博客集群任務(wù)管理系統(tǒng)SGE的簡(jiǎn)明教程注益。
SJM 一種HPC-SGE的任務(wù)組織與狀態(tài)監(jiān)控工具
- SJM簡(jiǎn)介
SJM全稱為Simple Job Manager，由StanfordBioinformatics團(tuán)隊(duì)負(fù)責(zé)開發(fā)與維護(hù)溯捆，它是一個(gè)用于管理在計(jì)算集群上運(yùn)行的一組相關(guān)作業(yè)的程序聊浅，是在sge-qsub的更上一層的抽象，具有如下特性：- 它提供了一種方便的方法來(lái)指定作業(yè)之間的依賴關(guān)系以及每個(gè)作業(yè)的資源需求（例如內(nèi)存现使、CPU內(nèi)核）
- 它監(jiān)視作業(yè)的狀態(tài)低匙，以便您可以知道整個(gè)組何時(shí)完成。如果任何作業(yè)失斕夹狻（例如顽冶，由于計(jì)算節(jié)點(diǎn)崩潰），SJM允許您恢復(fù)售碳，而無(wú)需重新運(yùn)行成功完成的作業(yè)
- 最后强重，SJM提供了一種可移植的方式绞呈，可以將作業(yè)提交給不同的作業(yè)調(diào)度器，如Sun網(wǎng)格引擎或平臺(tái)LSF
- 軟件安裝
# 注：軟件安裝前需要確認(rèn)是否安裝了Boost http://www.boost.org git clone https://github.com/StanfordBioinformatics/SJM.git cd SJM ./configure make sudo make install - SJM job中的屬性介紹
一個(gè)示例,任務(wù)以job_begin和job_end開始和結(jié)尾间景，任務(wù)名稱用name標(biāo)注佃声，內(nèi)存申請(qǐng)用memory標(biāo)注，任務(wù)命令用cmd標(biāo)注倘要，投遞隊(duì)列用queue標(biāo)注圾亏，：
或者用一下的形式代替：job_begin name jobA queue test.q time 1h memory 500m cmd echo "hello from job jobA" job_end
多行命令在一個(gè)任務(wù)中時(shí)可以如下編寫：job_begin name jobA sched_options -cwd -l vf=1G,p=1 -q test.q cmd echo "hello from job jobA" job_end
編寫多個(gè)任務(wù)之間的前后關(guān)系時(shí)，采用order屬性標(biāo)記封拧，如下：job_begin name jobB time 2d memory 1G queue test.q cmd_begin /home/lacroute/project/jobB_prolog.sh; /home/lacroute/project/jobB.sh; /home/lacroute/project/jobB_epilog.sh cmd_end job_end
order jobA before jobB - 編寫一個(gè)SJM的JOB并投遞
編寫任務(wù)：假設(shè)一個(gè)任務(wù)有A志鹃，B兩個(gè)部分，且A必須在B正常結(jié)束后才能執(zhí)行泽西，可復(fù)用代碼如下：
投遞任務(wù)：基于以上代碼寫在在test.job文件中曹铃，流程投遞命令如下：job_begin name jobA time 4h memory 3G queue standard project sequencing cmd /home/lacroute/project/jobA.sh job_end job_begin name jobB time 2d memory 1G queue extended cmd_begin /home/lacroute/project/jobB_prolog.sh; /home/lacroute/project/jobB.sh; /home/lacroute/project/jobB_epilog.sh cmd_end job_end order jobA before jobB log_dir /home/lacroute/project/logexport PATH=sjm的文件夾:$PATH # 投遞至后臺(tái) sjm test.job # 前臺(tái)投遞 sjm --interactive --log test.job.status.log test.job - 基于SJM的任務(wù)流的監(jiān)控
sjm投遞任務(wù)后（后臺(tái)投遞情況下），將會(huì)產(chǎn)生額外三個(gè)文件捧杉，其中*.status文件為每個(gè)job任務(wù)狀態(tài)記錄文件陕见，會(huì)伴隨更新，job狀態(tài)如下幾種味抖，注意及時(shí)查看輸出的log文件信息评甜。# 文件 test.job.status test.job.status.bak test.job.status.log # 狀態(tài) waiting 等待，未投遞 running 正在執(zhí)行 failed 運(yùn)行失敗 done 運(yùn)行成功 - 基于sjm-job 抽象的模板式任務(wù)編輯工具暢想
我們可以規(guī)定一套模板非竿，模塊使用makefile編寫蜕着，模塊組織使用sjm谋竖，實(shí)現(xiàn)流程模塊的高復(fù)用和高可用
其他類型的HPC任務(wù)流投遞工具
- 基于Kubernetes集群的ARGO
Argo Workflows是一個(gè)開源的容器原生工作流引擎红柱，用于在Kubernetes上編排并行作業(yè)”统耍可應(yīng)用與混合云等多種云計(jì)算集群的任務(wù)編排工具锤悄，高度滿足命令式編排和聲明式自動(dòng)化的特點(diǎn)。
基于argo創(chuàng)建任務(wù)流可參見博客ARGO-工作流部署與管理工具 - 基于Cromwell的WDL
WDL 是主要應(yīng)用與生物信息領(lǐng)域的流程定義工具嘉抒，需要依賴Cromwell引擎投遞任務(wù)零聚，但集群如果是SGE，其底層也是依賴QSUB投遞任務(wù)的些侍，單機(jī)節(jié)點(diǎn)下的配置文件如下：# Cromwell HTTP server settings include required(classpath("application")) webservice { #port = 8100 interface = 0.0.0.0 binding-timeout = 5s instance.name = "reference" } #database { # profile = "slick.jdbc.MySQLProfile$" ## db { # driver = "com.mysql.cj.jdbc.Driver" # url = "jdbc:mysql://host/cromwell?rewriteBatchedStatements=true" # user = "user" # password = "pass" # connectionTimeout = 5000 # } #} call-caching { enabled = true # In a multi-user environment this should be false so unauthorized users don't invalidate results for authorized users. invalidate-bad-cache-results = true } docker { hash-lookup { # Set this to match your available quota against the Google Container Engine API #gcr-api-queries-per-100-seconds = 1000 # Time in minutes before an entry expires from the docker hashes cache and needs to be fetched again #cache-entry-ttl = "20 minutes" # Maximum number of elements to be kept in the cache. If the limit is reached, old elements will be removed from the cache #cache-size = 200 # How should docker hashes be looked up. Possible values are "local" and "remote" # "local": Lookup hashes on the local docker daemon using the cli # "remote": Lookup hashes on docker hub and gcr #method = "remote" } } backend { default = Local providers { Local { actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory" config { run-in-background = true runtime-attributes = """ String? docker String? docker_user String? sge_mount """ submit = "${job_shell} ${script}" submit-docker = """ # make sure there is no preexisting Docker CID file rm -f ${docker_cid} # run as in the original configuration without --rm flag (will remove later) docker run \ --cidfile ${docker_cid} \ -i \ ${"--user " + docker_user} \ --entrypoint ${job_shell} \ -v ${cwd}:${docker_cwd}:delegated \ ${sge_mount} \ ${docker} ${docker_script} # get the return code (working even if the container was detached) rc=$(docker wait `cat ${docker_cid}`) # remove the container after waiting docker rm `cat ${docker_cid}` # return exit code exit $rc """ kill-docker = "docker kill `cat ${docker_cid}`" root = "cromwell-executions" filesystems { # For SFS backends, the "local" configuration specifies how files are handled. local { # Try to hard link (ln), then soft-link (ln -s), and if both fail, then copy the files. localization: [ "hard-link", "soft-link", "copy" ] # Call caching strategies caching { # When copying a cached result, what type of file duplication should occur. # For more information check: https://cromwell.readthedocs.io/en/stable/backends/HPC/#shared-filesystem duplication-strategy: [ "hard-link", "soft-link", "copy" ] # Strategy to determine if a file has been used before. # For extended explanation and alternative strategies check: https://cromwell.readthedocs.io/en/stable/Configuring/#call-caching hashing-strategy: "md5" # When true, will check if a sibling file with the same name and the .md5 extension exists, and if it does, use the content of this file as a hash. # If false or the md5 does not exist, will proceed with the above-defined hashing strategy. check-sibling-md5: false } } } # The defaults for runtime attributes if not provided. default-runtime-attributes { failOnStderr: false continueOnReturnCode: 0 } } } } } - nextflow
流程預(yù)定義隶症，可以直接下載模板后運(yùn)行，定制化程度不高岗宣，特殊需求下蚂会，需要手動(dòng)運(yùn)行命令分析，詳情參考生信流程大全-基于nextflow的nf-core - snakemake
某種程度上和WDL比較像耗式，通過(guò)模塊間的輸入輸出定義流程的DAG胁住。
