kubernetes admission webhook
原創(chuàng)請(qǐng)勿轉(zhuǎn)載
完成代碼示例: https://github.com/allenhaozi/webhook
環(huán)境準(zhǔn)備:
我是 MAC 本地開發(fā), 安裝docker后, 安裝kind, 本地啟動(dòng) k8s 集群
- go version v1.19.0+
- kind v0.17.0 go1.19.2 darwin/amd64
- kubebuilder v3.7.0
使用 kubebuilder 開發(fā)Operator 和 webhook server
初始化項(xiàng)目
kubebuilder init --domain github.com --repo github.com/allenhaozi/webhook
kubebuilder create api --group meta --version v1 --kind MetaWebHook
創(chuàng)建自己的CRD
kubebuilder create api --group meta --version v1 --kind MetaWebHook
創(chuàng)建完成后, 在對(duì)應(yīng)的 webhook/api/v1/metawebhook_types.go 增加自定義的值
// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.
// MetaWebHookSpec defines the desired state of MetaWebHook
type MetaWebHookSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
# add the following custom logic code
ServiceType string `json:"serviceType,omitempty"`
Database string `json:"database,omitempty"`
DatabaseSchema string `json:"databaseSchema,omitempty"`
TableFQN string `json:"tableFQN,omitempty"`
TableId string `json:"tableId,omitempty"`
}
構(gòu)建CRD并安裝
make manifests
make install
查看crd 安裝結(jié)果
$k get crds
NAME CREATED AT
metawebhooks.meta.github.com 2022-11-06T10:14:25Z
構(gòu)建完 CRD 以后, 構(gòu)建 對(duì)應(yīng)的webhook server
僅構(gòu)建 MutatingWebhookConfiguration 測(cè)試使用
kubebuilder create webhook --group meta --version v1 --kind MetaWebHook --defaulting
運(yùn)行完成后在根目錄 main.go 里面多了下面這寫代碼
if err = (&metav1.MetaWebHook{}).SetupWebhookWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create webhook", "webhook", "MetaWebHook")
os.Exit(1)
}
同時(shí)多了這個(gè)文件 api/v1/metawebhook_webhook.go
修改代碼: Default() 就是實(shí)現(xiàn) MutatingWebhookConfiguration 作用的函數(shù)
我們重寫了 TableId = "456"
即: 我們監(jiān)聽的所有 metawebhooks 資源的 TableId 都會(huì)被重寫為 456
package v1
import (
ctrl "sigs.k8s.io/controller-runtime"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/webhook"
"github.com/allenhaozi/alog"
)
// log is for logging in this package.
var metawebhooklog = logf.Log.WithName("metawebhook-resource")
func (r *MetaWebHook) SetupWebhookWithManager(mgr ctrl.Manager) error {
return ctrl.NewWebhookManagedBy(mgr).
For(r).
Complete()
}
// TODO(user): EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
//+kubebuilder:webhook:path=/mutate-meta-github-com-v1-metawebhook,mutating=true,failurePolicy=fail,sideEffects=None,groups=meta.github.com,resources=metawebhooks,verbs=create;update,versions=v1,name=mmetawebhook.kb.io,admissionReviewVersions=v1
var _ webhook.Defaulter = &MetaWebHook{}
// Default implements webhook.Defaulter so a webhook will be registered for the type
func (r *MetaWebHook) Default() {
metawebhooklog.Info("default", "name", r.Name)
alog.Pretty(r)
r.Spec.TableId = "456"
// TODO(user): fill in your defaulting logic.
}
本文重點(diǎn): webhook self signed
因?yàn)椴幌胍腩~外的組件管理CA證書, 比如 cert-manager
我們通過自簽名的方式來實(shí)現(xiàn)和api-server tls 的簽發(fā)
原理流程
-
實(shí)現(xiàn)自簽名的代碼, 自己生成三個(gè) ca 文件
- tls.key
- tls.crt
- ca.crt
tls.key 和 tls.crt 提供給 webhook server
ca.crt 更新到對(duì)應(yīng)的 MutatingWebhookConfiguration caBundle 字段上
通過自己實(shí)現(xiàn)一個(gè) controller watch MutatingWebhookConfiguration 的create和update 然后更新它
原理圖
webhook self signed architecture
上 code
MutatingWebhookConfiguration watch controller
package controllers
import (
"context"
"reflect"
"github.com/go-logr/logr"
admissionv1 "k8s.io/api/admissionregistration/v1"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/event"
"sigs.k8s.io/controller-runtime/pkg/predicate"
"github.com/allenhaozi/webhook/api/common"
)
// MetaWebHookReconciler reconciles a MetaWebHook object
type MutatingWebhookConfigurationReconciler struct {
client.Client
Scheme *runtime.Scheme
CaCert []byte
Log logr.Logger
}
// +kubebuilder:rbac:groups=admissionregistration.k8s.io,resources=mutatingwebhookconfigurations,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=admissionregistration.k8s.io,resources=mutatingwebhookconfiguration/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=admissionregistration.k8s.io,resources=mutatingwebhookconfiguration/finalizers,verbs=update
// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the MetaWebHook object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.13.0/pkg/reconcile
func (r *MutatingWebhookConfigurationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var m admissionv1.MutatingWebhookConfiguration
if err := r.Get(ctx, req.NamespacedName, &m); err != nil {
r.Log.Error(err, "got error")
return ctrl.Result{}, err
}
r.Log.Info("received mutatingwebhookconfiguration data", "MutatingWebhookConfiguration", req.NamespacedName)
if err := r.patchCaBundle(&m); err != nil {
r.Log.Error(err, "fail to patch CABundle to mutatingWebHookConfiguration")
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}
func (r *MutatingWebhookConfigurationReconciler) patchCaBundle(m *admissionv1.MutatingWebhookConfiguration) error {
ctx := context.Background()
current := m.DeepCopy()
for i := range m.Webhooks {
m.Webhooks[i].ClientConfig.CABundle = r.CaCert
}
if reflect.DeepEqual(m.Webhooks, current.Webhooks) {
r.Log.Info("no need to patch the MutatingWebhookConfiguration", "name", m.GetName())
return nil
}
if err := r.Patch(ctx, m, client.MergeFrom(current)); err != nil {
r.Log.Error(err, "fail to patch CABundle to mutatingWebHook", "name", m.GetName())
return err
}
r.Log.Info("finished patch MutatingWebhookConfiguration caBundle", "name", m.GetName())
return nil
}
// add
var filterByWebhookName = &predicate.Funcs{
CreateFunc: createPredicate,
UpdateFunc: updatePredicate,
}
func createPredicate(e event.CreateEvent) bool {
obj, ok := e.Object.(*admissionv1.MutatingWebhookConfiguration)
if !ok {
return false
}
if obj.GetName() == common.WebHookName {
return true
}
return false
}
func updatePredicate(e event.UpdateEvent) bool {
obj, ok := e.ObjectOld.(*admissionv1.MutatingWebhookConfiguration)
if !ok {
return false
}
if obj.GetName() == common.WebHookName {
return true
}
return false
}
// MutatingWebhookConfiguration
// SetupWithManager sets up the controller with the Manager.
func (r *MutatingWebhookConfigurationReconciler) SetupWithManager(mgr ctrl.Manager, l logr.Logger, caCert []byte) error {
return ctrl.NewControllerManagedBy(mgr).
For(&admissionv1.MutatingWebhookConfiguration{}).
WithEventFilter(filterByWebhookName).
Complete(
NewMutatingWebhookConfigurationReconciler(mgr, l, caCert),
)
}
func NewMutatingWebhookConfigurationReconciler(mgr ctrl.Manager, l logr.Logger, caCert []byte) *MutatingWebhookConfigurationReconciler {
r := &MutatingWebhookConfigurationReconciler{}
r.Client = mgr.GetClient()
r.Log = l
r.CaCert = caCert
return r
}
self signed certificate
package utils
import (
"crypto"
cryptorand "crypto/rand"
"crypto/rsa"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"log"
"math"
"math/big"
"os"
"path/filepath"
"time"
"github.com/pkg/errors"
"k8s.io/client-go/util/cert"
"k8s.io/client-go/util/keyutil"
)
var (
rsaKeySize = 2048
CertificateBlockType = "CERTIFICATE"
)
type CertContext struct {
// server.crt
Cert []byte
// server.key
Key []byte
SigningKey []byte
// ca.crt
SigningCert []byte
}
func GenerateCertAndCreate(namespaceName, serviceName, certDir string) (*CertContext, error) {
certContext := generateCert(namespaceName, serviceName)
// ca.crt
caCertFile := filepath.Join(certDir, "ca.crt")
if err := os.WriteFile(caCertFile, certContext.SigningCert, 0o644); err != nil {
return nil, errors.Errorf("Failed to write CA cert %v", err)
}
// server.key
keyFile := filepath.Join(certDir, "tls.key")
if err := os.WriteFile(keyFile, certContext.Key, 0o644); err != nil {
return nil, errors.Errorf("Failed to write key file %v", err)
}
// server.csr
certFile := filepath.Join(certDir, "tls.crt")
if err := os.WriteFile(certFile, certContext.Cert, 0o600); err != nil {
return nil, errors.Errorf("Failed to write cert file %v", err)
}
return certContext, nil
}
// reference: https://github.com/kubernetes/kubernetes/blob/v1.21.1/test/e2e/apimachinery/certs.go.
func generateCert(namespaceName, serviceName string) *CertContext {
signingKey, err := NewPrivateKey()
if err != nil {
log.Fatalf("Failed to create CA private key %v", err)
}
signingCert, err := cert.NewSelfSignedCACert(cert.Config{CommonName: "self-signed-k8s-cert"}, signingKey)
if err != nil {
log.Fatalf("Failed to create CA cert for apiserver %v", err)
}
key, err := NewPrivateKey()
if err != nil {
log.Fatalf("Failed to create private key for %v", err)
}
signedCert, err := NewSignedCert(
&cert.Config{
CommonName: serviceName + "." + namespaceName + ".svc",
AltNames: cert.AltNames{DNSNames: []string{serviceName + "." + namespaceName + ".svc"}},
Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
},
key,
signingCert,
signingKey,
)
if err != nil {
log.Fatalf("Failed to create cert%v", err)
}
keyPEM, err := keyutil.MarshalPrivateKeyToPEM(key)
if err != nil {
log.Fatalf("Failed to marshal key %v", err)
}
signingKeyPEM, err := keyutil.MarshalPrivateKeyToPEM(signingKey)
if err != nil {
log.Fatalf("Failed to marshal key %v", err)
}
c := &CertContext{
Cert: EncodeCertPEM(signedCert),
Key: keyPEM,
SigningCert: EncodeCertPEM(signingCert),
SigningKey: signingKeyPEM,
}
return c
}
// NewSignedCert creates a signed certificate using the given CA certificate and key
func NewSignedCert(cfg *cert.Config, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) {
serial, err := cryptorand.Int(cryptorand.Reader, new(big.Int).SetInt64(math.MaxInt64))
if err != nil {
return nil, err
}
if len(cfg.CommonName) == 0 {
return nil, errors.New("must specify a CommonName")
}
if len(cfg.Usages) == 0 {
return nil, errors.New("must specify at least one ExtKeyUsage")
}
certTmpl := x509.Certificate{
Subject: pkix.Name{
CommonName: cfg.CommonName,
Organization: cfg.Organization,
},
DNSNames: cfg.AltNames.DNSNames,
IPAddresses: cfg.AltNames.IPs,
SerialNumber: serial,
NotBefore: caCert.NotBefore,
NotAfter: time.Now().Add(time.Hour * 24 * 365 * 10).UTC(),
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: cfg.Usages,
}
certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &certTmpl, caCert, key.Public(), caKey)
if err != nil {
return nil, err
}
return x509.ParseCertificate(certDERBytes)
}
// NewPrivateKey creates an RSA private key
func NewPrivateKey() (*rsa.PrivateKey, error) {
return rsa.GenerateKey(cryptorand.Reader, rsaKeySize)
}
// EncodeCertPEM returns PEM-endcoded certificate data
func EncodeCertPEM(cert *x509.Certificate) []byte {
block := pem.Block{
Type: CertificateBlockType,
Bytes: cert.Raw,
}
return pem.EncodeToMemory(&block)
}
構(gòu)建鏡像
Dockerfile
FROM alpine:3.15.0
# Build the manager binary
ARG TARGETOS
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
WORKDIR /
COPY webhook-amd64 /webhook
USER 65532:65532
ENTRYPOINT ["/webhook"]
build.sh
set -x
VERSION=$1
GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o webhook-amd64 main.go
docker build -t allenhaozi/webhook.tar:v0.0.${VERSION} -f Dockerfile.local .
kind load docker-image allenhaozi/webhook.tar:v0.0.${VERSION} --name kind-dev
make deploy
k get pods
NAME READY STATUS RESTARTS AGE
controller-manager-7785bddd47-gkcpw 2/2 Running 0 6h2m
check example code
$cat config/samples/m.yaml
apiVersion: meta.github.com/v1
kind: MetaWebHook
metadata:
labels:
app.kubernetes.io/name: metawebhook
app.kubernetes.io/instance: metawebhook-sample
app.kubernetes.io/part-of: webhook
app.kuberentes.io/managed-by: kustomize
app.kubernetes.io/created-by: webhook
name: metawebhook-sample
spec:
# TODO(user): Add fields here
serviceType: "databaseService"
database: "salesforce"
databaseSchema: "default"
tableFQN: "naton"
tableId: "123"
提交測(cè)試yaml
$k apply -f config/samples/m.y
檢查提交的資源
$k get MetaWebHook metawebhook
-sample -oyaml
apiVersion: meta.github.com/v1
kind: MetaWebHook
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"meta.github.com/v1","kind":"MetaWebHook","metadata":{"annotations":{},"labels":{"app.kuberentes.io/managed-by":"kustomize","app.kubernetes.io/created-by":"webhook","app.kubernetes.io/instance":"metawebhook-sample","app.kubernetes.io/name":"metawebhook","app.kubernetes.io/part-of":"webhook"},"name":"metawebhook-sample","namespace":"default"},"spec":{"database":"salesforce","databaseSchema":"default","serviceType":"databaseService","tableFQN":"naton","tableId":"123"}}
creationTimestamp: "2022-11-09T12:08:49Z"
generation: 1
labels:
app.kuberentes.io/managed-by: kustomize
app.kubernetes.io/created-by: webhook
app.kubernetes.io/instance: metawebhook-sample
app.kubernetes.io/name: metawebhook
app.kubernetes.io/part-of: webhook
name: metawebhook-sample
namespace: default
resourceVersion: "6601135"
uid: 0324d4c2-2dc1-46a2-8ed2-37218f10085c
spec:
database: salesforce
databaseSchema: default
serviceType: databaseService
tableFQN: naton
tableId: "456"
tableId 從 123 修改 456 , 構(gòu)建成功
