Now the application is HTTPS client certificate authentication enabled.Positive and Negative TestTest without the valid cert.curl -k https://192.168.64.244:31463/date<html><head><title>400 No required SSL certificate was sent</title></head><body><center><h1>400 Bad Request</h1></center><center>No required SSL certificate was sent</center><hr><center>nginx/1.15.7</center></body></html>To test with the valid cert, let’s generate the cert signed by the CA first.Create a json file as below, save it as “clientRequest.json”{ "CN": "client-for-hello-server", "hosts": [ "" ], "key": { "algo": "rsa", "size": 2048 }}Generate the client cert with the client profile (as defined in the first paper).cd certscfssl gencert -ca=myca.pem -ca-key=myca-key.pem -config=ca-config.json -profile=client -hostname="127.0.0.1" clientRequest.json | cfssljson -bare hello-clientYou will havehello-client.pem, the public keyhello-client-key.pem, the private keyTest with these keys,curl -k –cert certshello-client.pem –key certshello-client-key.pem https://192.168.64.244:31463/datetime now: 03:36:36At this point, we implemented the HTTPS with client cert authentication for the non-https application using the K8s sidecar pattern.ICP PrometheusBy default, Prometheus doesn’t provide any HTTP/TLS capability..IBM Cloud Private, ICP, uses the sidecar technique to enable the HTTPS/TLS and client certificate authentication.Attached is part of the result of kubectl -n kube-system describe pods monitoring-prometheus-74c6d846d7-plb2nThe sidecar container, router, in the podsrouter: Container ID: docker://04b9191d822ddb7f14e063e264afaeb2299d6d846777d230b17ec7404f92bade Image: devcluster.icp:8500/ibmcom/icp-management-ingress:2.2.2 Image ID: docker-pullable://devcluster.icp:8500/ibmcom/icp-management-ingress@sha256:c6e8be6e465e69b0d9e045a78a42ec22ef9e79d1886dfce691b9e7f1e9738d6a Port: 8080/TCP Host Port: 0/TCP Command: /opt/ibm/router/entry/entrypoint.sh State: Running Started: Wed, 28 Nov 2018 15:16:22 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /opt/ibm/router/caCerts from monitoring-ca-certs (rw) /opt/ibm/router/certs from monitoring-certs (rw) /opt/ibm/router/conf from router-config (rw) /opt/ibm/router/entry from router-entry (rw) /opt/ibm/router/lua-scripts from lua-scripts-config-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-tvb2h (ro)The nginx.conf can be found by running kubectl -n kube-system describe cm monitoring-prometheus-router-nginx-configThe server block excerpt is listed as below,server { listen 8443 ssl default_server; ssl_certificate server.crt; ssl_certificate_key server.key; ssl_client_certificate /opt/ibm/router/caCerts/tls.crt; ssl_verify_client on; ssl_protocols TLSv1.2; # Modulo ChaCha20 cipher..ssl_ciphers EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:!EECDH+3DES:!RSA+3DES:!MD5; ssl_prefer_server_ciphers on;…Because of the ssl_verify_client setting, the client which needs to contact to Prometheus must, therefore, use the certificate signed by the CA cert.The location block, which redirects the traffic to the Prometheus, is shown below,location /federate { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_pass http://127.0.0.1:9090/prometheus/federate;}location /api/v1/series { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; if ($arg_match[] = "helm_release_info") { content_by_lua 'rewrite.write_release_response()'; } rewrite_by_lua 'rewrite.rewrite_query()'; proxy_pass http://127.0.0.1:9090/prometheus/api/v1/series;}location / { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host;rewrite_by_lua 'rewrite.rewrite_query()'; proxy_pass http://127.0.0.1:9090/prometheus/;}The API call to Prometheus will be redirected to the Prometheus container in the same pod.ConclusionThese two papers explore the sidecar pattern for HTTPS and client certificate authentication..With the knowledge, it is useful to understand the ICP Prometheus and to further extend Prometheus functionality.. More details