k8s 定制开发(kubebuilder)

kubebuilder介绍

需求与设计

之前讲过使用Traefik + etcd的内外网关打通的方案, 我们的服务在启动的时候可以通过框架直接注册到etcd中去, 那如果是其他框架或者语言, 比如 php, python, java 也想使用 Traefik+etcd这套方案这么办?

直接能想到的就是每个框架和语言都实现一个套注册到etcd的功能, 实现起来也并不难, 但是这就面临对别人服务的侵入性, 别人不一定愿意接受

现在的微服务开发，基本都基于容器部署, 而容器管理平台k8s也是当今容器编排工具的标准, k8s本身是知道当前系统中有哪些service服务的, 那能否通过感知service的变化，来把service信息动态写入的etcd喃?

那我们的解决方案就很简单了:

k8s service <---watch---- service operater ------> etcd

环境准备

go version v1.20.0+ docker version 17.03+. kubectl version v1.11.3+. Access to a Kubernetes v1.11.3+ cluster.

准备k8s集群

直接在云商哪儿创建一个托管集群

安装kubectl

我们下载一个kubectl命令到本地:

curl -LO "https://dl.k8s.io/release/v1.23.0/bin/windows/amd64/kubectl"

把kubectl copy到你的path路径下去就可以了

然后简单验证下:

$ kubectl get ns
NAME              STATUS   AGE
default           Active   111d
kube-node-lease   Active   111d
kube-public       Active   111d
kube-system       Active   111d
mpaas             Active   30d

安装 kubebuilder

Mac/Linux系统安装方法

curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

使用说明

关于kubebuilder的使用说明一定要阅读:

$ kubebuilder -h
CLI tool for building Kubernetes extensions and tools.

Usage:
  kubebuilder [flags]
  kubebuilder [command]

Examples:
The first step is to initialize your project:
    kubebuilder init [--plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]]

<PLUGIN KEYS> is a comma-separated list of plugin keys from the following table
and <PROJECT VERSION> a supported project version for these plugins.

                             Plugin keys | Supported project versions
-----------------------------------------+----------------------------
               base.go.kubebuilder.io/v3 |                          3
               base.go.kubebuilder.io/v4 |                          3
        declarative.go.kubebuilder.io/v1 |                       2, 3
 deploy-image.go.kubebuilder.io/v1-alpha |                          3
                    go.kubebuilder.io/v2 |                       2, 3
                    go.kubebuilder.io/v3 |                          3
                    go.kubebuilder.io/v4 |                          3
         grafana.kubebuilder.io/v1-alpha |                          3
      kustomize.common.kubebuilder.io/v1 |                          3
      kustomize.common.kubebuilder.io/v2 |                          3

For more specific help for the init command of a certain plugins and project version
configuration please run:
    kubebuilder init --help --plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]

Default plugin keys: "go.kubebuilder.io/v4"
Default project version: "3"


Available Commands:
  alpha       Alpha-stage subcommands
  completion  Load completions for the specified shell
  create      Scaffold a Kubernetes API or webhook
  edit        Update the project configuration
  help        Help about any command
  init        Initialize a new project
  version     Print the kubebuilder version

Flags:
  -h, --help                     help for kubebuilder
      --plugins strings          plugin keys to be used for this subcommand execution
      --project-version string   project version (default "3")

Use "kubebuilder [command] --help" for more information about a command.

创建项目

我们需要使用kubebuilder来为我们生成Operator开发的框架代码

kubebuilder 提供了一个 init命令用来初始化一个新的Operator工程目录，具体用法如下:

$ kubebuilder init -h
Initialize a new project including the following files:
  - a "go.mod" with project dependencies
  - a "PROJECT" file that stores project configuration
  - a "Makefile" with several useful make targets for the project
  - several YAML files for project deployment under the "config" directory
  - a "cmd/main.go" file that creates the manager that will run the project controllers

Usage:
  kubebuilder init [flags]

Examples:
  # Initialize a new project with your domain and name in copyright
  kubebuilder init --plugins go/v4 --domain example.org --owner "Your name"

  # Initialize a new project defining a specific project version
  kubebuilder init --plugins go/v4 --project-version 3


Flags:
      --domain string            domain for groups (default "my.domain")
      --fetch-deps               ensure dependencies are downloaded (default true)
  -h, --help                     help for init
      --license string           license to use to boilerplate, may be one of 'apache2', 'none' (default "apache2")
      --owner string             owner to add to the copyright
      --project-name string      name of this project
      --project-version string   project version (default "3")
      --repo string              name to use for go module (e.g., github.com/user/repo), defaults to the go package of the current working directory.
      --skip-go-version-check    if specified, skip checking the Go version

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution

关键参数说明:

--plugins, 指定生成代码的插件, 默认使用 "go.kubebuilder.io/v3"
--project-version 支持的项目版本, 有2,3 默认3
--repo module path, 就是 go mod init 指定的go module的名称
--domain, 组织名称, 用于API Group等
--owner, operater所有者, 一般填写开发者邮箱

然后开始初始化项目:

$ kubebuilder init --domain magedu.com --repo gitee.com/go-course/go12/skills/operator

INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.16.3 
go: downloading sigs.k8s.io/controller-runtime v0.16.3
go: downloading k8s.io/apimachinery v0.28.3
go: downloading k8s.io/client-go v0.28.3
go: downloading k8s.io/api v0.28.3
go: downloading k8s.io/apiextensions-apiserver v0.28.3
go: downloading k8s.io/component-base v0.28.3
INFO Update dependencies:
$ go mod tidy           
go: downloading github.com/onsi/gomega v1.27.10
go: downloading github.com/onsi/ginkgo/v2 v2.11.0
go: downloading go.uber.org/zap v1.25.0
go: downloading golang.org/x/tools v0.9.3
go: downloading github.com/benbjohnson/clock v1.3.0
Next: define a resource with:
$ kubebuilder create api

创建API

通过脚手架为我们提供的create api来创建 CRD 相关的Resource 和控制器:

$ kubebuilder create api -h
Scaffold a Kubernetes API by writing a Resource definition and/or a Controller.

If information about whether the resource and controller should be scaffolded
was not explicitly provided, it will prompt the user if they should be.

After the scaffold is written, the dependencies will be updated and
make generate will be run.

Usage:
  kubebuilder create api [flags]

Examples:
  # Create a frigates API with Group: ship, Version: v1beta1 and Kind: Frigate
  kubebuilder create api --group ship --version v1beta1 --kind Frigate

  # Edit the API Scheme

  nano api/v1beta1/frigate_types.go

  # Edit the Controller
  nano internal/controller/frigate/frigate_controller.go

  # Edit the Controller Test
  nano internal/controller/frigate/frigate_controller_test.go

  # Generate the manifests
  make manifests

  # Install CRDs into the Kubernetes cluster using kubectl apply
  make install

  # Regenerate code and run against the Kubernetes cluster configured by ~/.kube/config
  make run


Flags:
      --controller           if set, generate the controller without prompting the user (default true)
      --force                attempt to create resource even if it already exists
      --group string         resource Group
  -h, --help                 help for api
      --kind string          resource Kind
      --make make generate   if true, run make generate after generating files (default true)
      --namespaced           resource is namespaced (default true)
      --plural string        resource irregular plural form
      --resource             if set, generate the resource without prompting the user (default true)
      --version string       resource Version

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution

生成样例代码

kubebuilder create api --group traefik --version v1 --kind TraefikService
INFO Create Resource [y/n]                        
y
INFO Create Controller [y/n]                      
y
INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO api/v1/traefikservice_types.go               
INFO api/v1/groupversion_info.go                  
INFO internal/controller/suite_test.go            
INFO internal/controller/traefikservice_controller.go 
INFO Update dependencies:
$ go mod tidy           
INFO Running make:
$ make generate                
mkdir -p /Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin
test -s /Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/controller-gen && /Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/controller-gen --version | grep -q v0.13.0 || \
        GOBIN=/Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0
go: downloading sigs.k8s.io/controller-tools v0.13.0
go: downloading github.com/fatih/color v1.15.0
go: downloading k8s.io/api v0.28.0
go: downloading k8s.io/apimachinery v0.28.0
go: downloading github.com/gobuffalo/flect v1.0.2
go: downloading golang.org/x/net v0.14.0
go: downloading golang.org/x/text v0.12.0
/Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

样例生成完了后，我们会看到我们项目新增了一些文件:

.
|____cmd
| |____main.go
|____go.mod
|____bin
| |____controller-gen
|____config
| |____rbac
| | |____auth_proxy_service.yaml
| | |____leader_election_role_binding.yaml
| | |____role.yaml
| | |____leader_election_role.yaml
| | |____kustomization.yaml
| | |____role_binding.yaml
| | |____traefikservice_editor_role.yaml
| | |____service_account.yaml
| | |____auth_proxy_role.yaml
| | |____traefikservice_viewer_role.yaml
| | |____auth_proxy_client_clusterrole.yaml
| | |____auth_proxy_role_binding.yaml
| |____crd
| | |____kustomizeconfig.yaml
| | |____kustomization.yaml
| |____default
| | |____manager_auth_proxy_patch.yaml
| | |____kustomization.yaml
| | |____manager_config_patch.yaml
| |____samples
| | |____traefik_v1_traefikservice.yaml
| | |____kustomization.yaml
| |____manager
| | |____kustomization.yaml
| | |____manager.yaml
| |____prometheus
| | |____monitor.yaml
| | |____kustomization.yaml
|____Dockerfile
|____Makefile
|____internal
| |____controller
| | |____traefikservice_controller.go
| | |____suite_test.go
|____hack
| |____boilerplate.go.txt
|____go.sum
|____.golangci.yml
|____README.md
|____PROJECT
|____.dockerignore
|____.gitignore
|____api
| |____v1
| | |____groupversion_info.go
| | |____zz_generated.deepcopy.go
| | |____traefikservice_types.go

api/v1 目录下主要存放是我们API Object, 就是我们的Resource对象相关信息
config/crd 目录下是我们crd的描述文件, 我们需要把自定义资源(CRD)的描述信息注册给k8s时需要的
rbac 目录下存放着关于CRD资源的 role定义的样例文件(editor/viewer)
samples 目录下存放着 CRD的一个样例文件, 后面部署完成后可以直接编辑下 apply到k8s集群中去
controllers 目录下存放着我们所有的 Object 的Controller 代码

CRD开发

按照之前的设计，我们其实是没有必要定义CRD的, 下面关于CRD的定义和安装是出于教学演示目的, 如果你只想做项目，可以忽略这部分内容

CRD 设计

我们需要定义Traefik Service, 我们来看看Traefik Service一个service 实例定义:

<etcd_prefix>/<entry_point>/services/loadBalancer/servers/<index>/url   <url_value>

traefik etcd配置的前缀, provider配置时 有设置
services: 表示 web entrypoint的 services配置
loadBalancer: cmdb 服务loadBalancer配置
servers: loadBalancer 下的实例配置
0(变量): index

因此我们定义的Service需要有如下属性:

entrypoint name
service name
service url

作为k8s的CRD 必须是一个runtime.Object, 也就是为我们生成的TraefikService对象, 我们需要编辑的是TraefikServiceSpec对象

由于 Name已经在 ObjectMeta 有声明了, 因此我们只需要添加 entrypoint 和 url

修改资源定义: api/v1/traefikservice_types.go

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// TraefikServiceSpec defines the desired state of TraefikService
type TraefikServiceSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Foo is an example field of TraefikService. Edit traefikservice_types.go to remove/update
	Entrypoint string `json:"entrypoint"`
	URL        string `json:"url"`
}

// TraefikServiceStatus defines the observed state of TraefikService
type TraefikServiceStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// 是否被激活
	Active bool `json:"active,omitempty"`
	// 最新更新时间
	LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`
	// 如果更新失败, 失败的原因
	Message string `json:"message,omitempty"`
}

CRD代码生成

我们通过make 提供

manifests 重新生成修改后的 CRD定义描述
generate 重新生成代码

$ make manifests generate

/Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."

我们看到CRD的描述文件已经有变化了:

$ :: operator/config/crd ‹master*› » tree .
.
|____kustomizeconfig.yaml
|____kustomization.yaml
|____bases
| |____traefik.magedu.com_traefikservices.yaml

traefik.magedu.com_traefikservices.yaml的内容

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.13.0
  name: traefikservices.traefik.magedu.com
spec:
  group: traefik.magedu.com
  names:
    kind: TraefikService
    listKind: TraefikServiceList
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: TraefikService is the Schema for the traefikservices API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TraefikServiceSpec defines the desired state of TraefikService
            properties:
              entrypoint:
                description: Foo is an example field of TraefikService. Edit traefikservice_types.go
                  to remove/update
                type: string
              url:
                type: string
            required:
            - entrypoint
            - url
            type: object
          status:
            description: TraefikServiceStatus defines the observed state of TraefikService
            properties:
              active:
                description: 是否被激活
                type: boolean
              lastUpdateTime:
                description: 最新更新时间
                format: date-time
                type: string
              message:
                description: 如果更新失败, 失败的原因
                type: string
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

到此我们的CRD的描述文件已经生成ok, 剩下的就是把该文件注册到k8s集群内, 也就是我们所的安装CRD

安装CRD

脚手架使用

install: 安装CRD
uninstall: 卸载CRD

Deployment
  install          Install CRDs into the K8s cluster specified in ~/.kube/config.
  uninstall        Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.

提前准备好你的集群访问凭证: ~/.kube/config
kubebuilder 是通过SDK来操作的, 执行这个命令并不依赖你本地kubectl

执行如下命令安装:

$ make install
/Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
test -s /Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/kustomize || GOBIN=/Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@v5.2.1
go: downloading sigs.k8s.io/kustomize/kustomize/v5 v5.2.1
go: downloading sigs.k8s.io/kustomize/kyaml v0.15.0
go: downloading sigs.k8s.io/kustomize/api v0.15.0
go: downloading sigs.k8s.io/kustomize/cmd/config v0.12.0
go: downloading github.com/go-errors/errors v1.4.2
go: downloading k8s.io/kube-openapi v0.0.0-20230601164746-7562a1006961
go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: downloading github.com/xlab/treeprint v1.2.0
go: downloading gopkg.in/evanphx/json-patch.v5 v5.6.0
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
/Users/oldyu/Workspace/Golang/go-course-project/go12/skills/operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/traefikservices.traefik.magedu.com created

查看是否安装成功, 由于我的集群是腾讯托管的, 界面上可以直接看到,如下:

$ kubectl get crd
NAME                                 CREATED AT
traefikservices.traefik.magedu.com   2024-01-07T00:40:57Z

验证CRD

接下来我们验证通过CRD来创建一个 TraefikService的资源

准备CRD的YAML定义

我们参考samples下的样例, 补充spec相关参数

apiVersion: traefik.magedu.com/v1
kind: TraefikService
metadata:
  name: traefikservice-sample
spec:
  # TODO(user): Add fields here
  entrypoint: web
  url: https://www.baidu.com

创建我们的自定义资源:

# 创建资源
$ kubectl apply -f config/samples/traefik_v1_traefikservice.yaml 
traefikservice.traefik.magedu.com/traefikservice-sample created

# 查看资源
$ kubectl get TraefikService
NAME                    AGE
traefikservice-sample   35s

# 查看资源yaml
$ kubectl get TraefikService -o yaml
apiVersion: v1
items:
- apiVersion: traefik.magedu.com/v1
  kind: TraefikService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"traefik.magedu.com/v1","kind":"TraefikService","metadata":{"annotations":{},"labels":{"app.kubernetes.io/created-by":"operator","app.kubernetes.io/instance":"traefikservice-sample","app.kubernetes.io/managed-by":"kustomize","app.kubernetes.io/name":"traefikservice","app.kubernetes.io/part-of":"operator"},"name":"traefikservice-sample","namespace":"default"},"spec":{"entrypoint":"web","url":"https://www.baidu.com"}}
    creationTimestamp: "2024-01-07T00:42:42Z"
    generation: 1
    labels:
      app.kubernetes.io/created-by: operator
      app.kubernetes.io/instance: traefikservice-sample
      app.kubernetes.io/managed-by: kustomize
      app.kubernetes.io/name: traefikservice
      app.kubernetes.io/part-of: operator
    name: traefikservice-sample
    namespace: default
    resourceVersion: "3230760"
    uid: 9f74998f-9e37-4f28-81ab-95a628f96319
  spec:
    entrypoint: web
    url: https://www.baidu.com
kind: List
metadata:
  resourceVersion: ""

最后我们删除我们定义的资源

$ kubectl delete -f config/samples/traefik_v1_traefikservice.yaml 
traefikservice.traefik.magedu.com "traefikservice-sample" deleted

到此为止我们仅仅完成了对象的基本操作, 比如创建和删除, 但是与这个对象关联的业务逻辑代码我们还没编写,也就是说我们仅仅完成了对象的声明, 那如何开发对象的业务逻辑喃，着就是涉及到Crontroller了

Crontroller开发

什么是Crontroller? 他和 CRD之间又有啥纠葛, 且听我一一道来

Crontroller的原理

我们可以通过API Server声明一个资源对象(CRD)，实践上是声明了对象的期望状态, 是一个愿望, 比如期望副本数量为3个, Controller的核心逻辑就是让这个愿望实现, 并且实时Watch对象的变化, 一旦对象变化，我们就需要再次做出调整，让现实中的状态变成期望的状态

因此Crontroller 是个面向期望的编程模型, 我们声明的这个期望对象，就是API Object(K8s Runtime Object)

Crontroller 业务逻辑编写

Reconcile 函数是 Operator 的核心逻辑, 位于 controllers/traefikservice_controller.go 文件中, 我们修改他，添加我们的业务逻辑

// TraefikServiceReconciler reconciles a TraefikService object
type TraefikServiceReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// 每当traefikv1.TraefikService{}对象有变化时，我们就会收到一个请求
func (r *TraefikServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// 获取日志对象
	l := log.FromContext(ctx, "namespace", req.Namespace)

	// TODO(user): your logic here

	// 1.通过名称获取TraefikService对象, 并打印
	var obj traefikv1.TraefikService
	if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
		// 如果Not Found则表示该资源已经删除, 需要做删除处理
		if apierrors.IsNotFound(err) {
			l.Info("delete service ...")
			err = nil
		} else {
      l.Error(err, "unable to fetch TraefikService")
    }

		return ctrl.Result{}, err
	}

	l.Info("get TraefikService object",
		"name", obj.Name,
		"url", obj.Spec.URL,
		"entrypoint", obj.Spec.Entrypoint)

	// 2. 更加状态 排除已经同步完成的对象
	if obj.Status.Active {
		l.Info("traefik service is active, skip ...")
		return ctrl.Result{}, nil
	}

	// 3. 注册服务到Traefik service中, 比如写入到etcd provider中
	l.Info("set traefik service ...")

	// 4. 修改成功调整对象的状态
	obj.Status.Active = true
	obj.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}
	if err := r.Status().Update(ctx, &obj); err != nil {
		l.Info("unable to update status", "reason", err.Error())
	}

	return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *TraefikServiceReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&traefikv1.TraefikService{}).
		Complete(r)
}

部署 Controller

为了确保我们的CRD描述是最新的, 我们重新安装下

make manifests && make install

为了方便起见，我们将在本地运行 controller，当然您也可以将其部署到 Kubernetes 上运行

make run

验证 Controller

我们继续使用之前的样例, 看看Controller能否感觉对象的变化, 正常工作

# 创建资源
$ kubectl apply -f config/samples/traefik_v1_traefikservice.yaml 
traefikservice.traefik.magedu.com/traefikservice-sample created

我们通过controller的日志, 来确认Controller是否正常工作

...
2024-01-07T08:46:20+08:00       INFO    setup   starting manager
2024-01-07T08:46:20+08:00       INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-01-07T08:46:20+08:00       INFO    controller-runtime.metrics      Starting metrics server
2024-01-07T08:46:20+08:00       INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-01-07T08:46:20+08:00       INFO    Starting EventSource    {"controller": "traefikservice", "controllerGroup": "traefik.magedu.com", "controllerKind": "TraefikService", "source": "kind source: *v1.TraefikService"}
2024-01-07T08:46:20+08:00       INFO    Starting Controller     {"controller": "traefikservice", "controllerGroup": "traefik.magedu.com", "controllerKind": "TraefikService"}
2024-01-07T08:46:20+08:00       INFO    Starting workers        {"controller": "traefikservice", "controllerGroup": "traefik.magedu.com", "controllerKind": "TraefikService", "worker count": 1}
2024-01-07T08:46:52+08:00       INFO    get TraefikService object       {"controller": "traefikservice", "controllerGroup": "traefik.magedu.com", "controllerKind": "TraefikService", "TraefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "e69f4e7d-56bf-4a57-bde5-227181e0a9ce", "namespace": "default", "name": "traefikservice-sample", "url": "https://www.baidu.com", "entrypoint": "web"}
2024-01-07T08:46:52+08:00       INFO    set traefik service ... {"controller": "traefikservice", "controllerGroup": "traefik.magedu.com", "controllerKind": "TraefikService", "TraefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "e69f4e7d-56bf-4a57-bde5-227181e0a9ce", "namespace": "default"}
2024-01-07T08:46:52+08:00       INFO    get TraefikService object       {"controller": "traefikservice", "controllerGroup": "traefik.magedu.com", "controllerKind": "TraefikService", "TraefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "0d024e51-81cf-464d-b821-80e2fbf3f3f2", "namespace": "default", "name": "traefikservice-sample", "url": "https://www.baidu.com", "entrypoint": "web"}
2024-01-07T08:46:52+08:00       INFO    traefik service is active, skip ...     {"controller": "traefikservice", "controllerGroup": "traefik.magedu.com", "controllerKind": "TraefikService", "TraefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "0d024e51-81cf-464d-b821-80e2fbf3f3f2", "namespace": "default"}

Pod Controller

并不是什么时候我们都需要 CRD的, 比如我们只想Watch Pod的变化, 因此我们可以独立开发Controller即可

设置需要Watch的资源对象

// SetupWithManager sets up the controller with the Manager.
func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&v1.Pod{}).
		Complete(r)
}

资源状态业务逻辑

func (r *EndpointReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// 获取日志对象
	l := log.FromContext(ctx)

	// TODO(user): your logic here

	// 1.通过名称获取Pods对象, 并打印
	var obj v1.Pod
	if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
		// 如果Not Found则表示该资源已经删除, 需要做删除处理
		if apierrors.IsNotFound(err) {
			l.Info("delete Pods ...",
				"namespace", req.Namespace,
				"name", req.Name)
			err = nil
		} else {
			l.Error(err, "unable to fetch Pods")
		}
	}

	eps, _ := json.Marshal(obj)
	l.Info(string(eps))

	return ctrl.Result{}, nil
}

Manager加载Controller

controller开发完成后，需要注册给Manager, 才能给启动, 因此我们在main.go中完成该controller的加载

func main() {
	var metricsAddr string
	var enableLeaderElection bool
	var probeAddr string
	flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
	flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
	flag.BoolVar(&enableLeaderElection, "leader-elect", false,
		"Enable leader election for controller manager. "+
			"Enabling this will ensure there is only one active controller manager.")
	opts := zap.Options{
		Development: true,
	}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

	ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))

	// 默认配置
	options := ctrl.Options{
		Scheme:                 scheme,
		MetricsBindAddress:     metricsAddr,
		Port:                   9443,
		HealthProbeBindAddress: probeAddr,
		LeaderElection:         enableLeaderElection,
		LeaderElectionID:       "d18673dd.magedu.com",
	}

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), options)
	if err != nil {
		setupLog.Error(err, "unable to start manager")
		os.Exit(1)
	}

	if err = (&controllers.TraefikServiceReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "TraefikService")
		os.Exit(1)
	}
	//+kubebuilder:scaffold:builder

	// Pods Controller
	if err = (&controllers.PodReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "Pods")
		os.Exit(1)
	}

	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up health check")
		os.Exit(1)
	}
	if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up ready check")
		os.Exit(1)
	}

	setupLog.Info("starting manager")
	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
		setupLog.Error(err, "problem running manager")
		os.Exit(1)
	}
}

验证Pods Watch功能

下面是创建一个Nginx 的Deployment 观察到的Pod变化

1.6479577314173741e+09  INFO    controller.pod  nginx-86bd55b966-hwhtz  {"reconciler group": "", "reconciler kind": "Pod", "name": "nginx-86bd55b966-hwhtz", "namespace": "default", "namespace": "default", "labels": {"k8s-app":"nginx","pod-template-hash":"86bd55b966","qcloud-app":"nginx"}}
1.647957731418373e+09   INFO    controller.pod  172.16.0.69     {"reconciler group": "", "reconciler kind": "Pod", "name": "nginx-86bd55b966-hwhtz", "namespace": "default"}
1.647957731419115e+09   INFO    controller.pod  test    {"reconciler group": "", "reconciler kind": "Pod", "name": "nginx-86bd55b966-hwhtz", "namespace": "default", "pod_id": null}

基于次, 我们就可以把Pod容器里面的服务注册到注册中心去了

OrzR3 / go-course

k8s 定制开发(kubebuilder)

kubebuilder介绍

需求与设计

环境准备

准备k8s集群

安装kubectl

安装 kubebuilder

Mac/Linux系统安装方法

使用说明

创建项目

创建API

生成样例代码

CRD开发

CRD 设计

CRD代码生成

安装CRD

验证CRD

Crontroller开发

Crontroller的原理

Crontroller 业务逻辑编写

部署 Controller

验证 Controller

Pod Controller

设置需要Watch的资源对象

资源状态业务逻辑

Manager加载Controller

验证Pods Watch功能

参考

简介

发行版

贡献者

近期动态

OrzR3 / go-course .gitee-modal { width: 500px !important; }

k8s 定制开发(kubebuilder)

kubebuilder介绍

需求与设计

环境准备

准备k8s集群

安装kubectl

安装 kubebuilder

Mac/Linux系统安装方法

使用说明

创建项目

创建API

生成样例代码

CRD开发

CRD 设计

CRD代码生成

安装CRD

验证CRD

Crontroller开发

Crontroller的原理

Crontroller 业务逻辑编写

部署 Controller

验证 Controller

Pod Controller

设置需要Watch的资源对象

资源状态业务逻辑

Manager加载Controller

验证Pods Watch功能

参考

简介

发行版

贡献者

近期动态

搜索帮助

OrzR3 / go-course