Building your own kubernetes CRDs

Building your own kubernetes CRDsPongsatorn TonglairoumBlockedUnblockFollowFollowingJul 4Sharing knowledge on building my own kubernetes CRDs.

I will explain steps I use to develop CRDs using my Githook example.

I explain the reason why I create this CRDs in Build cloud native CI/CD build pipeline from GIT webhook.

Note:This article is for a beginner who what to learn how to build kubernetes CRDThe goal is to understand steps to make a CRD.

You might not be able to follow it step-by-step since it has some dependencies such as knative installedWhat you will learn:Kubernetes CRDs conceptHow to build kubernetes CRD using kubebuilder and golangWhy do I need a CRD?CRD stands for Custom Resource Definition.

It is a way to create your own kubernetes resource that work the same way as Kubernetes resource such as Pod or Deployment.

With custom resource, you can put the knowledge on how to manage your resource into kubernetes cluster and that is so powerful.

For example, Strimzi is a kubernetes CRD to manage and running kafka cluster on kubernetes.

It eliminates a lot of problem running your own kafka cluster by putting the knowledge to manage kafka cluster into the CRD controller itself.

CRDs controller reconciliation loopYou need a controller (in a container running on cluster) to manage your CRD resource.

The controller will run in loop to check if what described in the CRD (desired) is match the actual state in the real world.

For example, you want container A and B to run on the cluster.

If either A or B does not run, controller will create it for you.

This is a robust way to manage what you want in a resilient and flexible like distributed environment.

Githook example CRDThis CRD is called “GitHook”.

It defines git webhook events and a build pipeline.

GitHook controller will subscribe webhook events to git repo and when the events happen, it will run a build pipeline as defined in the CRD.

How “GitHook” CRD worksGitHook CRD controller’s job is:make sure that webhook is registered to git repo with incorrect information.

make sure that there is a service running and wait for the webhook events.

We use Knative service to receive that webhook since it is easy to implement and can scale to 0 when it is not in used.

Building CRD using kubebuilder1.

SetupInstall kubebuilder and kustomizekubebuilder is a tool to help you get started kubernetes CRDs very fast.

You just need to define your custom type and controller logic.

Read more.

kustomize is a tool kubebuilder use to produce a yaml that your CRD needs in order to work on kubernetesPlease follow installation step here.

Initial projectWe will create a go project named “githook-tutorial” with module support so it requires go version 1.

11+ and the folder must be outside GOPATH.

mkdir githook-tutorialcd githook-tutorialgo mod init gitlab.

com/pongsatt/githookkubebuilder init –domain my.

domain“gitlab.

com/pongsatt/githook” is go module name.

If you change to something else, you need to replace import path in the sample code.

“my.

domain” will be your resource’s API path.

You can change it as neededCreate your custom resourceThis step is to tell kubebuilder the kind of resource you need.

We will create a resource named “GitHook” in group called “tools”.

kubebuilder create api –group tools –version v1alpha1 –kind GitHookWhen it is done, you should see.

kubebuilder generated files and foldersInteresting paths:api/ : contains your resource information in go.

you will edit your resource type hereconfig/ : contains all kubernetes yaml files needed to run your controllerMakefile: contains all scripts needed to build and deploy your CRD2.

Define resource typeEdit “api/v1alpha1/githook_types.

go” to include our custom type.

See githook_types.

go for complete code.

Key notes:If we need our type to be validated when creating our resource via yaml, we need to put comment on that type.

For example, we need your gitProvider to be gitlab, github or gogs.

// +kubebuilder:validation:Enum=gitlab;github;gogs// GitProvider providers name of git providertype GitProvider stringNext step is to generate utility function such as deepcopy by running command:make generate3.

Implement controllerAt this point, we have our custom type ready to be consumed by our controller.

How controller works:Our controller named “GitHookReconciler” has a method “Reconcile” which will be called every time “GitHook” resource has been created, updated or deleted.

It will process each event one by one in order using queue.

When an error occurred, we will return error state so current event will be requeue and wait to be processed later.

Our controller will control 2 other resources:Knative service (to be able to process git events when occurred).

Git webhook (to be able to receive git events).

Create/Update Logic:Get GitHook information by name using provided k8s client.

// Reconcile main reconcile logicfunc (r *GitHookReconciler) Reconcile(req ctrl.

Request) (ctrl.

Result, error) {.

err := r.

Get(context.

Background(), req.

NamespacedName, sourceOrg).

Get Knative service that created by this resource.

If not exist, create new.

If exist but information update, updates it.

func (r *GitHookReconciler) reconcile(source *v1alpha1.

GitHook) error {.

ksvc, err := r.

reconcileWebhookService(source).

Register git webhook using Knative service URL given from previous step and keep returned ID in GitHook resource.

func (r *GitHookReconciler) reconcile(source *v1alpha1.

GitHook) error {.

hookID, err := r.

reconcileWebhook(source, hookOptions).

Update current GitHook resource with latest information// Reconcile main reconcile logicfunc (r *GitHookReconciler) Reconcile(req ctrl.

Request) (ctrl.

Result, error) {.

if err := r.

Update(context.

Background(), source); err != nil { .

}.

Delete Logic:When GitHook resource get deleted, we detect this by checking field “DeletionTimestamp” is not nil.

Then, we will delete our dependent resources which, in this case, knative service and git webhook by calling finalize method.

// Reconcile main reconcile logicfunc (r *GitHookReconciler) Reconcile(req ctrl.

Request) (ctrl.

Result, error) {.

if sourceOrg.

ObjectMeta.

DeletionTimestamp == nil { .

} else { if r.

hasFinalizer(source.

(*v1alpha1.

GitHook).

Finalizers) { reconcileErr = r.

finalize(source.

(*v1alpha1.

GitHook)) }}.

In case that we cannot delete knative service or git webhook, we will not allow our GitHook to be deleted by not deleting finalizer information from GitHook resource.

func (r *GitHookReconciler) finalize(source *v1alpha1.

GitHook) error { .

r.

removeFinalizer(source)}Code:Edit file “controllers/githook_controller.

go”Please see complete code githook_controller.

go.

Key Notes:Our controller needs permission to get, list, create and update another kubernetes resource.

We need to add comment annotations for kubebuilder to generate permission yaml for us.

// +kubebuilder:rbac:groups=tools.

pongzt.

com,resources=githooks,verbs=get;list;watch;create;update;patch;delete.

// Reconcile main reconcile logicfunc (r *GitHookReconciler) Reconcile(req ctrl.

Request) (ctrl.

Result, error) {.

4.

Update managerSince our controller uses other resource apis such as knative, we need to register those types in “main.

go”.

func init() { githookv1alpha1.

AddToScheme(scheme) servingv1alpha1.

AddToScheme(scheme) servingv1beta1.

AddToScheme(scheme) tektonv1alpha1.

AddToScheme(scheme) corev1.

AddToScheme(scheme) // +kubebuilder:scaffold:scheme}5.

Testing controllerYou can test or debug your controller on local machine by running command:make installmake runmake install : will generate CRDs yaml files and apply to your kubernetes clustermake run : will run test and run “main.

go”Your controller will start monitoring your resource and do the reconciliation loop.

6.

Deploy CRDs and controllerIt is the time you need your CRD to be usable on your cluster.

You need to run command below to deploy:make docker-build IMG=<your image registry>/controllermake docker-push IMG=<your image registry>/controller make deploymake docker-build will build docker image as your IMG variablemake docker-push will push docker image to registrymake deploy will produce all the yaml file and deploy to your clusterOptionally, you can release your yaml file for other to install your CRD easily using command:kustomize build config/default > release.

yamlConclusionIn this article, I cover only the important points to build a CRD.

The part I did not cover is:The code that run as Knative service webhookThe concept of resource owner (GitHook owns Knative service it created)The code for git api clientGitlab CI/CD used to build and release GitHook resourceThe main goal of this article is to share what I experienced building my CRD.

Hope you find it usefuls.

Thanks :D.

. More details

Leave a Reply