At Apigee, we're using Kubernetes to automate deployment, scaling, and management of containerized apps. While working on securing access to it, our goal was to use our existing single-sign-on solution. In the process, we learned a few things that we felt could be useful to other Kubernetes users.
Here, we’ll discuss how we used Cloud Foundry's UAA as an OpenID connect provider for Kubernetes authentication. If you’re not using UAA but you're using an OAuth2 provider for authentication, this post could be useful to you, too.
Note: This post provides background on our process and how we successfully wired things up. If this doesn’t interest you and you just want to know the steps required to use UAA (and possibly other OAuth 2.0 providers) as an OIDC provider for Kubernetes, please skip to the Cliffs notes section below.
Upon evaluating the different options Kubernetes offers for cluster authentication, there was only one that seemed plausible: the OpenID Connect (OIDC) authentication provider. Per the OpenID Connect spec, OIDC is a "simple identity layer on top of the OAuth 2.0 protocol" and it just so happens that UAA is an OAuth 2.0 provider with limited OIDC support. Even with only limited OIDC support, this seemed like as good a place to start as any.
Our first step was to see how far we could get by configuring Kubernetes to use our UAA server as an OIDC provider. We passed the following command-line options to the Kubernetes API server (based on the OpenID Connect Tokens section of the Kubernetes authentication guide):
- --oidc-issuer-url: this tells Kubernetes where your OIDC server is
- --oidc-client-id: this tells Kubernetes the OAuth client application to use
But during startup, the API server failed to start, and we saw errors like this:
Failed to fetch provider config, trying again in 3s: invalid character '<' looking for beginning of value.
After some digging in the Kubernetes sources and the go-oidc sources, we found out that upon start, the Kubernetes API server expects to find a document located at $OIDC_ISSUER_URL/.well-known/openid-configuration. What kind of file is this and what are its contents? After some Googling, we found out this document is an OpenID Provider metadata document and is used as part of OpenID connect discovery, something UAA itself does not support.
OpenID connect discovery
Instead of giving up, we decided to look into what it takes to implement OIDC discovery, starting with the $OIDC_ISSUER_URL/.well-known/openid-configuration. Reading the "obtaining OpenID provider configuration information" portion of the OIDC discovery specification, we learned that $OIDC_ISSUER_URL/.well-known/openid-configuration is used by OIDC clients to obtain the OpenID provider configuration. Once we understood the structure of the URL that Kubernetes was looking for, we needed to understand the structure of this document.
As expected, the OIDC discovery specification explains this in the OpenID provider metadata section. Since this post is not an OpenID Connect tutorial, I will instead point you to a few public OpenID providers for reference:
Based on the OIDC discovery specification and the various examples we found online from public OIDC providers, we felt confident that we could create an OpenID provider metadata document for our UAA server. That was our next step.
Note: Since UAA does not support OIDC discovery, we had to serve the OpenID provider metadata document ourselves; you will likely need to solve this as well.
JSON web tokens and signing
Once we had our OpenID provider metadata document served at $OIDC_ISSUER_URL/.well-known/openid-configuration, we restarted the API server. This time, there were no errors related to OIDC, and the API server started successfully. The next step was to get a token and attempt to authenticate to Kubernetes using said token. Of course, depending on your environment, how you get your token will change, but for UAA users, you could use the UAAC to do this:
# Set the uaac target (The UAA server location)
uaac target $OIDC_ISSUER_URL
# Get a user token from UAA
uaac token authcode get
# Print the uaac contexts
The last command will print out your UAAC contexts and one should match your target server. Once you find it, the token needed is the access_token property.
Once we have the token from UAA, we need to create/update our Kubernetes client (kubectl) context to contain our newly-retrieved token like so:
# Create a new kubectl cluster configuration
kubectl config set-cluster $CLUSTER_NAME --server=$K8S_SERVER_URL --certificate-authority=$K8S_CA_CRT
# Configure a context user (This user IS NOT the username used to authenticate to Kubernetes, that is in your token)
kubectl config set-context $CONTEXT_NAME --cluster=$CLUSTER_NAME --user=$USER_NAME
# Configure the context user to use the token we just retrieved
kubectl config set-credentials $USER_NAME --token=$TOKEN
# Configure kubectl to use the context we just created
kubectl config use-context $CONTEXT_NAME
Here’s an example:
Note: It's only a coincidence that we use kube-solo-secure for all names in the examples below. That is not a requirement and is done purely to make cleaning things up simpler.
kubectl config set-cluster kube-solo-secure --server=kube-solo-secure --certificate-authority=/tmp/ca.crt --embed-certs
kubectl config set-context kube-solo-secure --cluster=kube-solo-secure --user=kube-solo-secure
kubectl config set-credentials kube-solo-secure --token="$TOKEN"
kubectl config use-context kube-solo-secure
Each of these commands above should output a value of [cluster|context|user] "kube-solo-secure" set., except for the kubectl config use-context command, which should have output switched to context "kube-solo-secure". Once this was done, we were ready to see how much further this got us so we ran kubectl get pods and, unfortunately, we got this error: error: you must be logged in to the server (the server has asked for the client to provide credentials)
Looking into the API server logs, we saw this error: Unable to authenticate the request due to an error: [oidc: failed syncing KeySet: illegal base64 data at input byte 19, crypto/rsa: verification error] After a great deal of research and digging around, we found out the JSON web keys document, whose location is set via the jwks_uri in the OpenID Provider Metadata document, was invalid; that's when we ran into our first incompatibility with UAA's OIDC support.
UAA's incompatibility with OIDC
JSON Web Keys (JWK) are used to verify JSON Web Tokens (JWT); the structure of a JWS mandates that the modulus used to verify signatures is to be base64url encoded but the modules (the n property of the JWK provided by UAA) was only base64 encoded.
So UAA is not encoding JWK appropriately per the JWK specification. This led to us file a bug and come up with a workaround. Much like the need for us to host our own /.well-known/openid-configuration document alongside UAA, we also created a new version of the JWS file (/token_keys) at (/k8s_token_keys) and updated our OpenID provider metadata document to have the jwks_uri use the new document.
After this was up, we re-ran kubectl get pods and this time we got another error: JWT claims invalid: invalid claim value: 'iss'. expected=$ISSUER_URL, found=$ISSUER_URL/oauth/token., crypto/rsa: verification error (notice the extra /oauth/token).
Note: At this point I would like to point out that while progress was being made, we were beginning to think we would continue down this rabbit hole forever.
The good news: this error was easy to understand and, based on the OpenID provider metadata documentation, the iss claim value MUST MATCH the issuer value of the OpenID provider metadata document. Unfortunately, this is not something you can toggle within UAA, which led to a pull request.
The purpose of this PR was to get the ball rolling on fixing this officially; the PR contains the exact changes we made to our custom UAA server to fix this. Once we deployed the new version of UAA with the PR changes made, lo and behold: kubectl get pods worked as expected.
Building a custom version of UAA to help it implement OIDC just for Kubernetes authentication might seem like a bit much, not to mention that that alone is just one of a handful of steps that workaround UAA's lack of OIDC support. If that’s the case, you have two options:
- Wait until UAA officially supports OIDC
- Use the dex (a pluggable OIDC provider from the CoreOS folks) UAA support
The explanation above discusses how we got to a working deployment of UAA that’s being used for Kubernetes authentication via OIDC. To summarize the things that were required, here's a bulleted list of the steps:
- Patch UAA (using this PR: https://github.com/cloudfoundry/uaa/pull/425) and rebuild to avoid /oauth/token from being appended to your iss claim
- Create a version of $UAA_SERVER/token_keys that has the n properties base64url encoded instead of just base64 encoded
- Create an OpenID provider metadata document based on the OpenID provider metadata and the examples linked to above
- Serve your OpenID provider metadata and JWS documents at $UAA_SERVER/.well-known/openid-configuration and $UAA_SERVER/k8s_token_keys respectively (the latter URL is just an example; you can use any path you like as long as it matches the jwks_uri property in your OpenID provider metadata document)
- Create an OAuth client application in UAA that has the appropriate scope (openid)
- Update the Kubernetes API server options to have the --oidc-issuer-url option set to the $UAA_SERVER portion of the URLs mentioned in steps two and four
- Update the Kubernetes API server options to have the --oidc-client-id option set to the UAA OAuth client application created in step five
- Update the Kubernetes API server options for OIDC as needed (beyond the update options in steps six and seven)
In the end, our goals were met and we were able to successfully use our single-sign-on solution for Kubernetes authentication. While it would be ideal if UAA supported OIDC and we could just point Kubernetes to UAA and call it good, the steps above are easy to repeat, safe and have enabled us to get what we need quickly.