Kubernetes authentication via GitHub OAuth and DexAmet UmerovBlockedUnblockFollowFollowingJan 15Here’s a step-by-step guide for generating kubectl credentials using Dex, dex-k8s-authenticator and GitHub.
Unfortunately it’s not in Kubernetes vanillaMy name is Amet Umerov and I’m a DevOps Engineer at Preply.
Introduction to Kubernetes authWe use Kubernetes for creating dynamic environments for devs and QA.
So we want to provide them access to Kubernetes via Dashboard and CLI.
Kubernetes vanilla doesn’t support authentication for kubectl out of box unlike OpenShift.
In this configuration example, we use:dex-k8s-authenticator — a web application for generating kubectl configDex — OpenID Connect providerGitHub — because we use GitHub at our companyUnfortunately Dex can’t handle groups with Google OIDC, so if you want to use groups, try another provider.
Without groups you can’t create group-based RBAC policies.
Here is a flow of how Kubernetes authorization works:The Authorization processUser initiates a login request in the dex-k8s-authenticator (login.
k8s.
example.
com)dex-k8s-authenticator redirects the request to Dex (dex.
k8s.
example.
com)Dex redirects to the GitHub authorization pageGitHub encrypts the corresponding information and passes it back to DexDex forwards this information to dex-k8s-authenticatorUser gets the OIDC token from GitHubdex-k8s-authenticator adds the token to kubeconfigkubectl passes the token to KubeAPIServerKubeAPIServer returns the result to kubectlUser gets the information from kubectlPrerequisitesSo, we have already installed Kubernetes cluster (k8s.
example.
com) and HELM.
Also we have GitHub with organization name (super-org).
If you don’t have HELM, you can easily install it.
Go to the GitHub organization Settings page, (https://github.
com/organizations/super-org/settings/applications) and create a new Authorized OAuth App:The GitHub settings pageFill the fields with your values:Homepage URL: https://dex.
k8s.
example.
comAuthorization callback URL: https://dex.
k8s.
example.
com/callbackBe careful with links, trailing slashes are important.
Save theClient ID and Client secret generated by GitHub in safe place (we use Vault for storing our secrets):Client ID: 1ab2c3d4e5f6g7h8Client secret: 98z76y54x32w1Prepare your DNS records for subdomains login.
k8s.
example.
com and dex.
k8s.
example.
com and SSL certificates for Ingress.
Create SSL certificates:cat <<EOF | kubectl create -f -apiVersion: certmanager.
k8s.
io/v1alpha1kind: Certificatemetadata: name: cert-auth-dex namespace: kube-systemspec: secretName: cert-auth-dex dnsNames: – dex.
k8s.
example.
com acme: config: – http01: ingressClass: nginx domains: – dex.
k8s.
example.
com issuerRef: name: le-clusterissuer kind: ClusterIssuer—apiVersion: certmanager.
k8s.
io/v1alpha1kind: Certificatemetadata: name: cert-auth-login namespace: kube-systemspec: secretName: cert-auth-login dnsNames: – login.
k8s.
example.
com acme: config: – http01: ingressClass: nginx domains: – login.
k8s.
example.
com issuerRef: name: le-clusterissuer kind: ClusterIssuerEOFkubectl describe certificates cert-auth-dex -n kube-systemkubectl describe certificates cert-auth-login -n kube-systemYour ClusterIssuer le-clusterissuer should already exist, if you haven’t done it you can easily create it via HELM:helm install –namespace kube-system -n cert-manager stable/cert-managercat << EOF | kubectl create -f -apiVersion: certmanager.
k8s.
io/v1alpha1kind: ClusterIssuermetadata: name: le-clusterissuer namespace: kube-systemspec: acme: server: https://acme-v02.
api.
letsencrypt.
org/directory email: k8s-admin@example.
com privateKeySecretRef: name: le-clusterissuer http01: {}EOFKubeAPIServer setupYou need to provide OIDC configuration for the kubeAPIServer as below and update cluster:kops edit cluster.
kubeAPIServer: anonymousAuth: false authorizationMode: RBAC oidcClientID: dex-k8s-authenticator oidcGroupsClaim: groups oidcIssuerURL: https://dex.
k8s.
example.
com/ oidcUsernameClaim: emailkops update cluster –yeskops rolling-update cluster –yesIn our case we use kops for cluster provisioning but it works the same for other clusters.
Dex and dex-k8s-authenticator setupFor connecting Dex you should have a Kubernetes certificate and key.
Let’s obtain from the master:sudo cat /srv/kubernetes/ca.
{crt,key}—–BEGIN CERTIFICATE—–AAAAAAAAAAABBBBBBBBBBCCCCCC—–END CERTIFICATE———-BEGIN RSA PRIVATE KEY—–DDDDDDDDDDDEEEEEEEEEEFFFFFF—–END RSA PRIVATE KEY—–Clone the dex-k8s-authenticator repo:git clone git@github.
com:mintel/dex-k8s-authenticator.
gitcd dex-k8s-authenticator/You can setup a values file very flexible, HELM charts are available on GitHub.
Dex will not work with default variables.
Create the values file for Dex:cat << EOF > values-dex.
ymlglobal: deployEnv: prodtls: certificate: |- —–BEGIN CERTIFICATE—– AAAAAAAAAAABBBBBBBBBBCCCCCC —–END CERTIFICATE—– key: |- —–BEGIN RSA PRIVATE KEY—– DDDDDDDDDDDEEEEEEEEEEFFFFFF —–END RSA PRIVATE KEY—–ingress: enabled: true annotations: kubernetes.
io/ingress.
class: nginx kubernetes.
io/tls-acme: "true" path: / hosts: – dex.
k8s.
example.
com tls: – secretName: cert-auth-dex hosts: – dex.
k8s.
example.
comserviceAccount: create: true name: dex-auth-saconfig: | issuer: https://dex.
k8s.
example.
com/ storage: # https://github.
com/dexidp/dex/issues/798 type: sqlite3 config: file: /var/dex.
db web: http: 0.
0.
0.
0:5556 frontend: theme: "coreos" issuer: "Example Co" issuerUrl: "https://example.
com" logoUrl: https://example.
com/images/logo-250×25.
png expiry: signingKeys: "6h" idTokens: "24h" logger: level: debug format: json oauth2: responseTypes: ["code", "token", "id_token"] skipApprovalScreen: true connectors: – type: github id: github name: GitHub config: clientID: $GITHUB_CLIENT_ID clientSecret: $GITHUB_CLIENT_SECRET redirectURI: https://dex.
k8s.
example.
com/callback orgs: – name: super-org teams: – team-red staticClients: – id: dex-k8s-authenticator name: dex-k8s-authenticator secret: generatedLongRandomPhrase redirectURIs: – https://login.
k8s.
example.
com/callback/envSecrets: GITHUB_CLIENT_ID: "1ab2c3d4e5f6g7h8" GITHUB_CLIENT_SECRET: "98z76y54x32w1"EOFAnd for dex-k8s-authenticator:cat << EOF > values-auth.
ymlglobal: deployEnv: proddexK8sAuthenticator: clusters: – name: k8s.
example.
com short_description: "k8s cluster" description: "Kubernetes cluster" issuer: https://dex.
k8s.
example.
com/ k8s_master_uri: https://api.
k8s.
example.
com client_id: dex-k8s-authenticator client_secret: generatedLongRandomPhrase redirect_uri: https://login.
k8s.
example.
com/callback/ k8s_ca_pem: | —–BEGIN CERTIFICATE—– AAAAAAAAAAABBBBBBBBBBCCCCCC —–END CERTIFICATE—–ingress: enabled: true annotations: kubernetes.
io/ingress.
class: nginx kubernetes.
io/tls-acme: "true" path: / hosts: – login.
k8s.
example.
com tls: – secretName: cert-auth-login hosts: – login.
k8s.
example.
comEOFInstall Dex and dex-k8s-authenticator:helm install -n dex –namespace kube-system –values values-dex.
yml charts/dexhelm install -n dex-auth –namespace kube-system –values values-auth.
yml charts/dex-k8s-authenticatorCheck it (Dex should returns code 400, and dex-k8s-authenticator — code 200):curl -sI https://dex.
k8s.
example.
com/callback | head -1HTTP/2 400curl -sI https://login.
k8s.
example.
com/ | head -1HTTP/2 200RBAC configurationCreate ClusterRole for your group, in our case with read-only permissions:cat << EOF | kubectl create -f -apiVersion: rbac.
authorization.
k8s.
io/v1kind: ClusterRolemetadata: name: cluster-read-allrules: – apiGroups: – "" – apps – autoscaling – batch – extensions – policy – rbac.
authorization.
k8s.
io – storage.
k8s.
io resources: – componentstatuses – configmaps – cronjobs – daemonsets – deployments – events – endpoints – horizontalpodautoscalers – ingress – ingresses – jobs – limitranges – namespaces – nodes – pods – pods/log – pods/exec – persistentvolumes – persistentvolumeclaims – resourcequotas – replicasets – replicationcontrollers – serviceaccounts – services – statefulsets – storageclasses – clusterroles – roles verbs: – get – watch – list – nonResourceURLs: ["*"] verbs: – get – watch – list – apiGroups: [""] resources: ["pods/exec"] verbs: ["create"]EOFCreate ClusterRoleBinding configuration:cat <<EOF | kubectl create -f -apiVersion: rbac.
authorization.
k8s.
io/v1beta1kind: ClusterRoleBindingmetadata: name: dex-cluster-auth namespace: kube-systemroleRef: apiGroup: rbac.
authorization.
k8s.
io kind: ClusterRole name: cluster-read-allsubjects: – kind: Group name: "super-org:team-red"EOFNow you are ready to start testing.
TestsGo to login page (https://login.
k8s.
example.
com) and sign in with your GitHub account:Login pageLogin page redirected to GitHubFollow the instructions for create kubectl configAfter copy-pasting commands from the login webpage you can use kubectl with your cluster:kubectl get poNAME READY STATUS RESTARTS AGEmypod 1/1 Running 0 3dkubectl delete po mypodError from server (Forbidden): pods "mypod" is forbidden: User "amet@example.
com" cannot delete pods in the namespace "default"And it works! All users with GitHub account in our organization can read resources and exec into pods, but have no write permissions.
Stay tuned and subscribe to our blog, we will publish new articles soon :).