spark on k8s 基礎(chǔ)鏡像的構(gòu)建

背景

這是跑spark on k8s任務(wù)的基礎(chǔ)鏡像酌心，用來(lái)指明executor pod的基礎(chǔ)鏡像

構(gòu)建步驟

• git clone spark特定的版本（加入是3.0.1版本）凌停，克隆完后杂伟，執(zhí)行一下命令進(jìn)行構(gòu)建,構(gòu)建出包含kubernetes模塊的可運(yùn)行包：

spark 3.x兼容hadoop cdh版本,處理沖突

git cherry-pick 8e8afb3a3468aa743d13e23e10e77e94b772b2ed
./dev/make-distribution.sh --name 2.6.0-cdh5.13.1 --pip --tgz -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests

  
- 安裝并加入必要的jar包
按照[lzoCodec](https://stackoverflow.com/questions/23441142/class-com-hadoop-compression-lzo-lzocodec-not-found-for-spark-on-cdh-5),安裝native-lzo library(用來(lái)支持lzo),     
把包含libhadoop.so的目錄下的文件復(fù)制到assembly/target/scala-2.12/jars/hadoop_native    
把包含libgplcompression.so的目錄下的文件復(fù)制到assembly/target/scala-2.12/jars/native    
配置環(huán)境變量

```bash
ENV SPARK_DIST_CLASSPATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
ENV LD_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
ENV JAVA_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native

• 修改鏡像代碼為

  # distribution, the docker build command should be invoked from the top level directory
  # of the Spark distribution. E.g.:
  # docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .
  
  RUN set -ex && \
      sed -i 's/http:\/\/deb.\(.*\)/https:\/\/deb.\1/g' /etc/apt/sources.list && \
      apt-get update && \
      ln -s /lib /lib64 && \
      apt install -y bash tini libc6 libpam-modules krb5-user libnss3 && \
      apt-get install liblzo2-dev -y && \
      mkdir -p /opt/spark && \
      mkdir -p /opt/spark/examples && \
      mkdir -p /opt/spark/work-dir && \
      touch /opt/spark/RELEASE && \
      rm /bin/sh && \
      ln -sv /bin/bash /bin/sh && \
      echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
      chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
      rm -rf /var/cache/apt/*
  
  COPY jars /opt/spark/jars
  COPY bin /opt/spark/bin
  COPY sbin /opt/spark/sbin
  COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
  COPY examples /opt/spark/examples
  COPY kubernetes/tests /opt/spark/tests
  COPY data /opt/spark/data
  
  ENV SPARK_HOME /opt/spark
  
  ENV SPARK_DIST_CLASSPATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  
  ENV LD_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  
  ENV JAVA_LIBRARY_PATH=$SPARK_HOME/jars/native:$SPARK_HOME/jars/hadoop_native
  
  RUN ln -s $SPARK_HOME/jars/hadoop-lzo-0.4.15-cdh5.13.1.jar $SPARK_HOME/jars/hadoop-lzo.jar
  WORKDIR /opt/spark/work-dir
  RUN chmod g+w /opt/spark/work-dir
  
  ENTRYPOINT [ "/opt/entrypoint.sh" ]
  

• 構(gòu)建包含k8s的鏡像,執(zhí)行如下命令：

  ./bin/docker-image-tool.sh -t spark-on-k8s-v3.0.1-cdh-2.6.0-5.13.1 build
  ## 按需進(jìn)行修改鏡像標(biāo)簽
  docker tag spark:spark-on-k8s-v3.0.1-cdh-2.6.0-5.13.1 xxx.xxx.xxx/xxx/spark-on-k8s:v3.0.1-cdh-2.6.0-5.13.1
  

任務(wù)鏡像的構(gòu)建

背景

這是用來(lái)跑spark on k8s任務(wù)的driver端的鏡像

構(gòu)建步驟

• 按照任務(wù)要求進(jìn)行鏡像增加
  注意對(duì)于spark on k8s client 在dockerfile中需配置

  ENV HADOOP_CONF_DIR=/opt/hadoop/conf
  
  RUN echo '\nexport SPARK_LOCAL_HOSTNAME=${POD_IP}' >> /path/to/spark/conf/spark-env.sh
  

  這樣在driver通行的過(guò)程中就不會(huì)出現(xiàn)executor連接不上driver端的情況，原因是因?yàn)閐river端以pod的名字作為host，而exeuctor直接訪問(wèn)該host是訪問(wèn)不了的,具體參考spark on k8s 與spark on k8s operator的對(duì)比

• 配置spark-default.conf
  在/path/to/spark/conf/spark-default.conf配置：

  spark.kubernetes.namespace                              dev
  spark.kubernetes.authenticate.driver.serviceAccountName lijiahong
  spark.kubernetes.authenticate.serviceAccountName        lijiahong
  ## 注意這里是之前構(gòu)建的spark-on-k8s的基礎(chǔ)鏡像，如果是以cluster形式運(yùn)行，則driver和executor的鏡像分開(kāi)配置
  ## spark.kubernetes.driver.container.image 
  ## spark.kubernetes.executor.container.image  
  spark.kubernetes.container.image                        xxx.xxx.xxx./xxx/spark-on-k8s:v3.0.1-cdh-2.6.0-5.13.1
  spark.kubernetes.container.image.pullSecrets            regsecret
  spark.kubernetes.file.upload.path                       hdfs://tmp
  spark.kubernetes.container.image.pullPolicy             Always
  

• 構(gòu)建鏡像

  docker build -f Dockerfile --pull -t "xxx/xxx/spark-on-k8s:xxx" .
  

• 提交任務(wù)的時(shí)候設(shè)置POD_IP
  如以下yaml文件：

  apiVersion: v1
  kind: Pod
  metadata:
  name: spark-on-k8s-demo
  labels:
    name: spark-on-k8s-demo
  spec:
   containers:
    - name: spark-on-k8s-demo
      image: xxx/xxx/spark-on-k8s:xxx
      imagePullPolicy: Always
      env:
      - name: POD_IP
        valueFrom:
          fieldRef:
              fieldPath: status.podIP
      - name: NODE_IP
        valueFrom:
          fieldRef:
              fieldPath: status.hostIP
   imagePullSecrets:
    - name: regsecret
   restartPolicy: Never
  

至此，spark on k8s構(gòu)建完畢