Automating Kubeflow Deployment with Helm and AWS CDK
Part 2: Configuring Cluster Authentication
Part 1: CI/CD setup and getting started with Helm + CDK
Part 2: Configuring Cluster Authentication
In the previous part of this series, we set up an automated pipeline using CDK Pipelines to deploy an EKS Cluster and install cert-manager and Istio. At the end, we deployed a minimal nginx service and validated that it was accessible via our Istio ingress gateway.
We are now almost ready to start installing the various components of Kubeflow; but first, we must set up authentication on our cluster to ensure that only authenticated users are allowed to access the Kubeflow UI and APIs.
We will use Dex, an open-source identity service, and an OIDC Auth Service to build an end-to-end authentication flow that will handle all of the incoming traffic to our cluster.
Installing Dex
To install Dex on EKS, we take an approach similar to the one used to install the Istio ingress gateway in the previous article: we create a new Helm chart that extends the official Dex chart. Specifically, this new Helm chart will allow us to integrate the Dex service into the Istio service mesh installed earlier.
We will call this new Helm chart dex-istio
helm create dex-istioWe mark the official Dex Helm chart as a dependency by adding it to the Chart.yaml file:
Then we add the following virtual-service.yaml file to the templates/ directory. This creates an Istio VirtualService resource that we can use to route traffic to the Dex service.
A VirtualService resource must be associated with one or more Gateway resources. This template pulls the list of Gateway resources from the values.yaml file. Later, when we install Kubeflow, we will have a single Gateway that handles all traffic coming into our Kubeflow installation; but for now, we can use the sample Gateway created at the end of the previous article.
To reference the sample Gateway, we add the following to values.yaml :
With this Helm chart created, we can add the installation of Dex to the CDK application:
After Dex is installed on the cluster, we can navigate to the Dex login page at the following URL (where {{ALB_DNS_NAME}} is the public DNS name of the AWS Application Load Balancer associated with the Istio ingress gateway):
{{ALB_DNS_NAME}}/dex/authHere, we will be greeted with a Dex error indicating that the provided Client ID is invalid.
Configuring Client Credentials
In order for a client application to use Dex for authentication, the client app must provide a Client ID and Client Secret pair that is recognized by Dex. To start out, we will simply configure Dex to accept a static set of Client Credentials.
This same approach is demonstrated in the official Kubeflow manifests repository; however, in this case, the client credentials are stored directly in GitHub. To avoid this, we will use AWS Secrets Manager to securely store the Client ID and Client Secret.
To make AWS Secrets Manager secrets available to the services running on the EKS cluster, we will need to set up the AWS Secrets and Configuration Provider (ASCP) as specified in this guide: https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html
To install the ASCP from within our CDK application, we first install the publicly available Secrets Store CSI Driver Helm chart as specified in Install the Secrets Store CSI Driver; then, we add the Kubernetes resources defined in https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
For brevity, I have not included the source code for the installation in this article, but you can find it in the GitHub repository: secrets-provider-deployment.ts
With the ASCP installed, we will now add a new Secret to AWS Secrets Manager. We can do this directly within the DexDeployment class in the CDK application:
This will create a Secret with two key/value pairs: The key CLIENT_ID will have the value kubeflow-oidc-authservice and the key CLIENT_SECRET will have a value generated at runtime by CloudFormation. This ensures that the secret value is never stored in source control.
In order to allow the Dex service to access this secret, we create a new Kubernetes Service Account and grant it the necessary permissions:
Next, we need to modify the dex-istio Helm chart to create a new SecretProviderClass resource. This resource allows us to sync our AWS Secrets Manager Secrets as Kubernetes Secrets. Check out Sync as Kubernetes Secret for more details. The SecretProviderClass template for our client credentials secret will look like this:
The following section is added to values.yaml to provide the values for the above template:
- On line 2, we use YAML Anchors to store a reference to the name of the
SecretProviderClassresource. This allows us to reference this value elsewhere in thevalues.yamlfile (we’ll see exactly how shortly). - On line 6, we provide an empty string as the value of
secretProviderClass.clientCredentialsSecret.awsSecret. Within the CDK application, we override this value to provide the actual name of the AWS Secrets Manager secret being created.
Now, when installing the Dex Helm chart, we pass in the name of the AWS Secret containing the client credentials. Also, we provide the name of the newly created service account, and set dex.serviceAccount.create to false to indicate that we want to use an existing service account instead of creating a new one.
Finally, we need to update the values.yaml file of the dex-istio Helm chart again to mount the client credentials secret as a volume, add the client credentials to the Dex service’s environment variables, and add the client ID and client secret pair as a valid client of the service:
Notice on line 8 above, we use the YAML anchor defined earlier to reference the name of the SecretProviderClass that is used to provide the client credentials secret.
After deploying all of these changes, we can test that the client credentials are setup correctly by navigating to the Dex service URL with the following query string parameters set (the values for client ID and secret can be retrieved from AWS Secrets Manager console):
+----------------+-------------------+
| Parameter Name | Parameter Value |
+----------------+-------------------+
| client_id | {{CLIENT_ID}} |
| client_secret | {{CLIENT_SECRET}} |
| redirect_uri | /login/oidc |
| scope | openid |
| response_type | code |
+----------------+-------------------+With these parameters set, we will be met with a Dex login page, indicating that the client credentials were accepted.
At this point, however, there are no valid user credentials that we can enter here. We will set up an example user with an email address and a password next.
Configuring Users
Dex can integrate with many popular identity providers like Google, GitHub, and Microsoft. For now though, we will just configure Dex to allow a static email address and password that we can use to log in.
We will use AWS Secrets Manager secrets again to securely store our example user’s password. When setting up the client credentials earlier, we allowed the CDK to generate a random secure string for the Client Secret value when running new Secret() . This approach will not work for creating the user’s password. For one, we want to know what the user’s password is so that we can enter it in the login screen; and more importantly, Dex requires that we only store a hash of the actual password. When a user enters their password on the login page, it is hashed and compared with the stored hash.
For these reasons, we need to first manually create the hashed user password secret through the AWS console, and then reference this secret from within the CDK application to allow the Dex service to access it.
To generate a hash of the desired password, we run the following command, where {{PASSWORD}} is the actual password value:
echo {{PASSWORD}} | htpasswd -BinC 10 admin | cut -d: -f2We now store this value as a secret by navigating to the AWS Secrets Manager console and selecting “Store New Secret”
We will select “Other type of secret” as the “Secret Type” and enter a Key/Value pair where the key is "PASSWORD" and the value is the hashed password value.
After manually creating this secret, we can reference it from within the CDK application using the Secret.fromSecretCompleteArn() function and the newly-created secret’s ARN like so:
const userPasswordSecret = Secret.fromSecretCompleteArn(
this, // construct scope
"UserPasswordSecret", // construct id
{{SECRET_ARN}} // secret ARN
);Now, we will update the dex-istio Helm chart to include the new user password secret in the SecretProviderClass resource alongside the client credentials secret from earlier. The resulting SecretProviderClass will look this:
Notice on line 33, the jmesPath.path attribute is given a value of PASSWORD . This value must match the Key portion of the Key/Value pair created through the AWS Secrets Manager console.
In values.yaml we add the required new values to the secretProviderClass section:
And the following section within the dex.config section configures Dex to accept the new hashed password for a user with email address user@example.com
The last step is to modify the DexDeployment class to grant the Dex service account permissions to read the user password secret and to provide the value for secretProviderClass.staticPasswordSecret.awsSecret when installing the dex-istio Helm chart.
You can find the completed code for this class here: dex-deployment.ts
Now, we can test that everything is working as expected by navigating back to the Dex login page and entering user@example.com as the Email Address, and the original un-hashed password value as the Password:
If the credentials are entered correctly, we will be met with an HTTP 404 error indicating that the /login/oidc page does not exist. This is where the OIDC AuthService comes in.
Installing OIDC AuthService
We now have a login page that can be accessed using a specific set of client credentials, and a working email address and password that can be used to authenticate through Dex.
Now, we need a way to make sure that all web traffic entering into our EKS cluster is first rerouted to Dex for authentication. We will use OIDC AuthService to accomplish this. Unfortunately there is currently no official Helm chart for this service, but we can create one fairly easily by reusing the manifests present in the Kubeflow manifests repository at https://github.com/kubeflow/manifests/tree/master/common/oidc-authservice/base
You can find the completed OIDC AuthService Helm chart in my GitHub repository at charts/oidc-authservice, so I will not go over every template defined in this chart. I will, however, discuss a few key resources.
Creating the Envoy Filter
An Envoy Filter is a custom Kubernetes resource defined by the Istio service mesh that was installed in the previous article. We use Envoy Filters to configure the Envoy Proxies which handle all of the incoming and outgoing network traffic in our service mesh.
To ensure all incoming traffic is authenticated, we create an Envoy Filter using the External Authorization HTTP Filter, which forwards each incoming request to a specified service, which determines whether the request is allowed, or if the caller must first authenticate before their request can be fulfilled.
The template for the EnvoyFilter resource in our oidc-authservice Helm chart looks like this:
A few things to note in the template above:
- On lines 6–10 we specify the
workloadSelectorfor our Envoy Filter. This is a set of labels that determines to which services this filter will be applied. In our case, we simply want to apply this filter to the Istio ingress gateway created in the previous article, since this is the gateway through which all traffic enters the service mesh. The exact labels will be provided via Helm values - On line 27 we provide the URI to the service that is used to determine whether or not the request is authorized. This is the OIDC auth service that we will be deploying next.
To validate that the Envoy Filter is working, we can apply it to the cluster and then try navigating back to the sample service from earlier at the DNS name of the ALB.
Now, we should be met with a 403 Forbidden error indicating that we are not authorized to view the page.
Installing the Auth Service
With an Envoy Filter appropriately redirecting unauthenticated incoming requests, we now need to create the service to handle those requests. To install the Auth Service, we copy the remaining manifests from kubeflow/manifests/blob/master/common/oidc-authservice/base into our Helm chart. The only real difference is in the way we handle configurations and secrets.
We list the default config settings in values.yaml like so:
And then we store them in a ConfigMap
The Auth Service also needs the client ID and client secret created earlier, so that it can access Dex. We provide these client credentials to the Auth Service using AWS Secrets Manager and a SecretProviderClass as we did earlier.
Once all of the templates are added to the Helm chart, we create an AuthServiceDeployment class in the CDK app similar to the DexDeployment class. Instead of creating a new secret to store the client credentials, however, this class accepts an existing secret as input through AuthServiceDeploymentProps
When creating an instance of AuthServiceDeployment we pass the client credentials secret that was created by the Dex deployment:
Once the Auth Service is deployed and running, we can test the cluster authentication setup end-to-end by navigating to the sample service again. This time we are successfully redirected to the Dex login page.
Upon entering the correct email address and password, we are directed back to the sample service, and greeted with the nginx welcome page
If we leave this page in the browser, and then navigate back, you will notice that we are not prompted for credentials again. This is because the Auth Service stores an authservice_session cookie after we login for the first time. This allows us to access the sample service freely for the duration of our session.
There are a lot of moving pieces in this process. The diagram below provides a high-level look at the journey of a single user request through the authentication process:
Wrapping up Part 2
In the first article in this series, we created a CI/CD pipeline to deploy our EKS cluster and install an Istio service mesh. This time, we setup authentication to ensure that all clients must provide a valid email address and password before they are able to access the services running in the service mesh.
We added the deployment of Dex and OIDC Auth Service to the existing CDK application and CI/CD pipeline. We stored all of the sensitive values involved in the authentication process securely in AWS Secrets Manager, and accessed them as Kubernetes Secrets; and, aside from the necessary manual creation of the sample user password, we were able to fully automate all secret storage and retrieval.
With all of this in place, we are now ready to deploy Kubeflow in part 3 of this series.
All of the source code for the completed CDK application can be found here: https://github.com/gnovack/kubeflow-helm-cdk
References:
