一. 準備工作與注意事項

1. 部署的項目情況

(1) 業(yè)務(wù)架構(gòu)及服務(wù)(dubbo, spring cloud)
(2) 第三方服務(wù)工扎，例如mysql, redis, zookeeper, eruke, mq
(3) 服務(wù)之間怎么通信
(4) 資源消耗：硬件資源遭商，帶寬

2. 項目部署涉及相關(guān)k8s資源

(1) 使用namespace 進行不同項目隔離宪巨，或者隔離不同環(huán)境( test, prod, dev)
(2) 無狀態(tài)應(yīng)用(deployment)
(3) 有狀態(tài)應(yīng)用(statefulset, pv, pvc)
(4) 發(fā)布暴露外部訪問( Service, ingress)
(5) 存儲一些數(shù)據(jù)(secret, configmap)

3. 項目的基礎(chǔ)鏡像

4. 編排部署(鏡像為交付物)

https://github.com/lizhenliang/tomcat-java-demo
https://github.com/lizhenliang/php-demo
代碼倉庫下都會有一個Dockerfile 鏡像構(gòu)建文件

(1). 持續(xù)集成與交付

項目構(gòu)建(java)，CI/CD 環(huán)境這個階段自動完成（代碼拉取->代碼編譯構(gòu)建->鏡像打包 -> 推送到鏡像倉庫）

(2). 編寫yaml文件

編寫yaml文件对竣， 使用這個鏡像

5. 持續(xù)部署

kubectl -> yaml -> 鏡像倉庫拉取鏡像 -> Service(集群內(nèi)部訪問)/Ingress暴露外部用戶訪問

二. 部署Java項目

1. NFS網(wǎng)絡(luò)存儲準備

安裝NFS服務(wù),每個node節(jié)點都安裝nfs客戶端：

# yum install nfs-utils -y
# cat /etc/exports
/ifs/kubernetes *(rw,no_root_squash)

# mkdir /ifs/kubernetes -p
# systemctl start nfs

2. 配置NFS動態(tài)卷供給PV

下載三個yaml文件：
https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client/deploy
class.yaml 動態(tài)創(chuàng)建PV
deployment.yaml 創(chuàng)建控制及pod等資源
rbac.yaml deployment.yaml部署的pod時，需要訪問api-server權(quán)限請求k8s的PV&PVC資源
注意： deployment.yaml中定義連接NFS的地址及目錄

3. 部署私有鏡像倉庫harbor

http://www.reibang.com/p/7ca6c59f9882
注意：私有鏡像倉庫-harbor 使用http訪問必須在每個node做信任,并重啟docker

# cat /etc/docker/daemon.json
{
  "registry-mirrors": ["http://bc437cce.m.daocloud.io"],
  "insecure-registries":["http://10.40.6.165"]
}

4. java代碼拉取

用一個node節(jié)點clone java代碼：

# yum install git -y 
# git clone https://github.com/lizhenliang/tomcat-java-demo.git
# cd tomcat-java-demo/
# ll
total 24
drwxr-xr-x 2 root root    34 Jun 15 11:40 db
-rw-r--r-- 1 root root   148 Jun 15 11:40 Dockerfile
-rw-r--r-- 1 root root 11357 Jun 15 11:40 LICENSE
-rw-r--r-- 1 root root  1930 Jun 15 11:40 pom.xml
-rw-r--r-- 1 root root    89 Jun 15 11:40 README.md
drwxr-xr-x 3 root root    18 Jun 15 11:40 src

要用容器化部署自己的項目榜配，一般在你項目代碼創(chuàng)建一個Dockerfile文件否纬，使用Dockerfile構(gòu)建項目鏡像。

# cat Dockerfile 
FROM lizhenliang/tomcat 
LABEL maintainer www.ctnrs.com
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 

5. maven和java環(huán)境準備

java是動態(tài)語言蛋褥，需要構(gòu)建war包或jar包临燃，這構(gòu)建需要maven和java環(huán)境

# tar xvf apache-maven-3.6.1-bin.tar.gz 
# tar xvf jdk-8u181-linux-x64.tar.gz 
#  mv apache-maven-3.6.1 /usr/local/services/
# mv jdk1.8.0_181 /usr/local/services/
# cat /etc/profile.d/java.sh
JAVA_HOME=/usr/local/services/jdk1.8.0_181
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH

# source /etc/profile
# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

# ln -sv /usr/local/services/apache-maven-3.6.1/bin/mvn /usr/bin/mvn

6. 項目代碼編譯

編譯完成后會生產(chǎn)一個target 目錄，此目錄會有一個war包烙心。

# cd tomcat-java-demo/
# mvn clean package
# ll target/
total 17840
drwxr-xr-x 5 root root       95 Jun 15 14:45 classes
drwxr-xr-x 3 root root       25 Jun 15 14:45 generated-sources
drwxr-xr-x 4 root root       37 Jun 15 14:46 ly-simple-tomcat-0.0.1-SNAPSHOT
-rw-r--r-- 1 root root 18265402 Jun 15 14:46 ly-simple-tomcat-0.0.1-SNAPSHOT.war
drwxr-xr-x 2 root root       28 Jun 15 14:46 maven-archiver
drwxr-xr-x 3 root root       35 Jun 15 14:45 maven-status

7. 構(gòu)建項目鏡像并推送至鏡像倉庫

通過Dockefile構(gòu)建項目鏡像

# cat Dockerfile 
FROM lizhenliang/tomcat 
LABEL maintainer www.ctnrs.com
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war

# docker build -t 10.40.6.165/project/java-demo:v1 .   ##docker build -t 鏡像倉庫推送地址
# docker login 10.40.6.165
# docker push 10.40.6.165/project/java-demo:v1    ##推到到鏡像倉庫

8. 編寫部署項目yaml文件

yaml部署資源流程:
tomcat：
deployment
service
ingress

mysql：
statefuleset
healess service
pv,pvc(storageclass PV 自動供給 )

yaml文件：
namespace.yaml
deployment.yaml
ingress.yaml
service.yaml
mysql.yaml
registry-pull-secret.yaml
這些yaml文件可以統(tǒng)一寫到一個yaml, 使用'---'分隔即可膜廊。

9. 創(chuàng)建項目資源

(1). 創(chuàng)建命名空間

# cat namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: test

#kubectl create -f namespace.yaml
# kubectl get ns -n test

(2). 創(chuàng)建鏡像倉庫Secret用戶驗證

拉取鏡像需要用戶驗證，先創(chuàng)建kubernetes 登錄鏡像倉庫的Secret用戶驗證淫茵，注意拉取鏡像的yaml配置文件中Secret name字段爪瓜，要與此創(chuàng)建的一致

# kubectl create secret --help
# kubectl create secret docker-registry --help
     ....
Usage:
kubectl create secret docker-registry NAME --docker-username=user --docker-password=password --docker-email=email

指定Secret創(chuàng)建到 test命名空間，因為我們的項目創(chuàng)建在test命名空間

# kubectl create secret docker-registry registry-pull-secret --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@test.com --docker-server=10.40.6.165 -n test

# kubectl get secret -n test
NAME                   TYPE                                  DATA   AGE
default-token-vqllw    kubernetes.io/service-account-token   3      30m
registry-pull-secret   kubernetes.io/dockerconfigjson        1      69s

(3). 創(chuàng)建項目pod

# cat deployment.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: tomcat-java-demo
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      project: www
      app: java-demo
  template:
    metadata:
      labels:
        project: www
        app: java-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: tomcat
        image: 10.40.6.165/project/java-demo:v1 
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        resources:
          requests:
            cpu: 0.5
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20

# kubectl create -f deployment.yaml
# kubectl get pod -n test

(4). 創(chuàng)建 service

# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: tomcat-java-demo
  namespace: test
spec:
  selector:
    project: www
    app: java-demo
  ports:
  - name: web
    port: 80
    targetPort: 8080

# kubectl create -f service.yaml
# kubectl get svc -n test

(5). 創(chuàng)建ingress規(guī)則

# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat-java-demo 
  namespace: test
spec:
  rules:
    - host: java.ctnrs.com
      http:
        paths:
        - path: /
          backend:
            serviceName: tomcat-java-demo 
            servicePort: 80
# kubectl create -f ingress.yaml 
# kubectl get ingress -n test
# kubectl get pods,svc,ing -n test

10. 測試訪問

# kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS   AGE    IP            NODE          NOMINATED NODE
nginx-ingress-controller-74cbc544f6-9v5qv   1/1     Running   0          2d3h   172.17.59.5   10.40.6.213   <none>
nginx-ingress-controller-74cbc544f6-qklwh   1/1     Running   0          2d3h   172.17.31.5   10.40.6.210   <none>

# kubectl get ing -n test
NAME               HOSTS            ADDRESS   PORTS   AGE
tomcat-java-demo   java.ctnrs.com             80      10m

綁定域名到ingress-nginx的節(jié)點IP上訪問：
10.40.6.210 java.ctnrs.com
http://java.ctnrs.com

11. 創(chuàng)建數(shù)據(jù)庫service匙瘪、pod铆铆、PV和PVC

# cat mysql.yaml 
apiVersion: v1
kind: Service
metadata:
  name: java-demo-mysql
  namespace: test
  labels:
    project: java-demo
    app: mysql
spec:
  ports:
  - port: 3306
    name: java-demo-mysql
  clusterIP: None
  selector:
    project: java-demo
    app: mysql

---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: db
  namespace: test
spec:
  selector:
    matchLabels:
      project: java-demo
      app: mysql
  serviceName: "java-demo-mysql"
  template:
    metadata:
      labels:
        project: java-demo
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5.7 
        env: 
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
        ports: 
        volumeMounts:
        - mountPath: "/var/lib/mysql"
          name: mysql-data
  volumeClaimTemplates:
  - metadata:
      name: mysql-data 
    spec:
      accessModes: ["ReadWriteMany"]
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 2Gi 

# kubectl create -f mysql.yaml
# kubectl get svc,pods -n test

# kubectl get pv,pvc -n test
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS          REASON   AGE
persistentvolume/pvc-03054e78-8f53-11e9-8287-005056b614b3   2Gi        RWX            Delete           Bound    test/mysql-data-db-0   managed-nfs-storage            14m

NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
persistentvolumeclaim/mysql-data-db-0   Bound    pvc-03054e78-8f53-11e9-8287-005056b614b3   2Gi        RWX            managed-nfs-storage   14m

12. 數(shù)據(jù)庫配置

將tables_ly_tomcat.sql sql文件導入進去蝶缀，并修改數(shù)據(jù)庫連接

# scp  db/tables_ly_tomcat.sql 10.40.6.201:~   ##將文件拷貝到master節(jié)點
# kubectl cp /root/tables_ly_tomcat.sql db-0:/ -n test   ## 將sql文件cp到db-0容器中
# kubectl exec -it db-0 -n test bash
root@db-0:/# mysql -uroot -p123456
mysql> source /tables_ly_tomcat.sql
Query OK, 1 row affected (0.01 sec)

Database changed
Query OK, 0 rows affected (0.02 sec)

Query OK, 0 rows affected (0.00 sec)
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.01 sec)
mysql> use test
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| user           |
+----------------+
1 row in set (0.00 sec)
mysql> desc user;
+-------+--------------+------+-----+---------+----------------+
| Field | Type         | Null | Key | Default | Extra          |
+-------+--------------+------+-----+---------+----------------+
| id    | int(11)      | NO   | PRI | NULL    | auto_increment |
| name  | varchar(100) | NO   |     | NULL    |                |
| age   | int(3)       | NO   |     | NULL    |                |
| sex   | char(1)      | YES  |     | NULL    |                |
+-------+--------------+------+-----+---------+----------------+
4 rows in set (0.02 sec)

登錄項目容器測試mysql容器域名：
域名格式：pod_name.service_name.namespace

# kubectl exec -it tomcat-java-demo-5f87cd895d-4zszj -n test bash
[root@tomcat-java-demo-5f87cd895d-4zszj tomcat]# ping -c 1 db-0.java-demo-mysql.test
PING db-0.java-demo-mysql.test.svc.cluster.local (172.17.59.6) 56(84) bytes of data.

# kubectl get pod -n test -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP            NODE          NOMINATED NODE
db-0                                1/1     Running   0          20m    172.17.59.6   10.40.6.213   <none>

修改項目連接數(shù)據(jù)庫配置文件 src/main/resources/application.yml

# cat src/main/resources/application.yml
server:
  port: 8080
spring:
  datasource:
    url: jdbc:mysql://db-0.java-demo-mysql.test:3306/test?characterEncoding=utf-8
    username: root
    password: 12345
    driver-class-name: com.mysql.jdbc.Driver
  freemarker:
    allow-request-override: false
    cache: true
    check-template-location: true
    charset: UTF-8
    content-type: text/html; charset=utf-8
    expose-request-attributes: false
    expose-session-attributes: false
    expose-spring-macro-helpers: false
    suffix: .ftl
    template-loader-path:
      - classpath:/templates/

更新代碼后更新鏡像，并推送到鏡像倉庫

# docker build -t 10.40.6.165/project/java-demo:v2 .
# docker push 10.40.6.165/project/java-demo:v2

修改deployment.yaml 項目鏡像版本為：10.40.6.165/project/java-demo:v2
滾動更新項目：kubectl apply -f deployment.yaml
然后訪問站點添加美女.

添加美女.png

今晚翻盤哪個美女.png

三. 部署PHP

部署跟java基本一致薄货，只是代碼不需要編譯構(gòu)建
clone代碼 -> 制作代碼鏡像 -> 代碼鏡像推送至鏡像倉庫->編寫yaml配置文件 ->創(chuàng)建資源對象

1. clone代碼

git clone https://github.com/lizhenliang/php-demo.git

2. 構(gòu)建項目鏡像并推送至鏡像倉庫

這里的數(shù)據(jù)庫就用上邊java項目創(chuàng)建的數(shù)據(jù)庫翁都，修改php代碼連接數(shù)據(jù)庫的配置文件：

php代碼連接數(shù)據(jù)庫的配置文件
# cat php-demo/wp-config.php
      ....
/** MySQL主機 */
define('DB_HOST', 'db-0.java-demo-mysql.test');
     ....


代碼目錄php-demo中的Dockfile文件：
# cat Dockerfile
FROM lizhenliang/nginx-php:latest
MAINTAINER www.ctnrs.com
ADD . /usr/local/nginx/html

構(gòu)建項目鏡像,并推送到鏡像倉庫
# docker build -t 10.40.6.165/project/php-demo:1.0 .
# docker push 10.40.6.165/project/php-demo:1.0 

3. 編寫yaml配置文件并創(chuàng)建資源對象

命名空間test 在java 項目已經(jīng)創(chuàng)建，這里不再創(chuàng)建

(1). 創(chuàng)建pod

# cat deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: php-demo
  namespace: test
spec:
  replicas: 2
  selector:
    matchLabels:
      project: www
      app: php-demo
  template:
    metadata:
      labels:
        project: www
        app: php-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: nginx
        image: 10.40.6.165/project/php-demo:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        resources:
          requests:
            cpu: 0.5
            memory: 256Mi
          limits:
            cpu: 1
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /index.php
            port: 80
          initialDelaySeconds: 6
          timeoutSeconds: 20

# kubectl create -f deployment.yaml
# kubectl get pod -n test

(2). 創(chuàng)建service

# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: php-demo
  namespace: test
spec:
  selector:
    project: www
    app: php-demo
  ports:
  - name: web
    port: 80
    targetPort: 80

# kubectl create -f service.yaml
# kubectl get svc -n test

(3). 創(chuàng)建ingress規(guī)則

# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: php-demo
  namespace: test
spec:
  rules:
    - host: php.ctnrs.com
      http:
        paths:
        - path: /
          backend:
            serviceName: php-demo
            servicePort: 80

# kubectl create -f ingress.yaml
# kubectl  get ing -n test

(4). 配置數(shù)據(jù)庫

php項目與java項目同用一個數(shù)據(jù)庫test, test庫中有同用一個user表谅猾，這里將之前java項目到user表刪除掉

# kubectl exec -it db-0 bash  -n test
root@db-0:/# mysql -uroot -p123456
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.00 sec)

mysql> use test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| user           |
+----------------+
1 row in set (0.00 sec)

mysql> drop table user;
Query OK, 0 rows affected (0.01 sec)

(5). 驗證

綁定域名柄慰，瀏覽器訪問

# kubectl get ing -n test
NAME               HOSTS            ADDRESS   PORTS   AGE
php-demo           php.ctnrs.com              80      14m
tomcat-java-demo   java.ctnrs.com             80      8h

本地綁定hosts:
10.40.6.210 php.ctnrs.com

訪問: http://php.ctnrs.com