IRSA Mutating Admission Webhook for Kubernetes

This webhook implements IAM Roles for Service Accounts (IRSA) for Kubernetes clusters, allowing pods to assume Vultr IAM roles using projected service account tokens.

Features

• Automatically injects Vultr credentials configuration into pods
• Compatibility with AWS SDK
• Uses projected service account tokens with custom audience
• Supports multiple containers and init containers
• Follows security best practices
• No external dependencies beyond Kubernetes API

How It Works

When a pod is created, the webhook:

1. Extracts the ServiceAccount name from the pod spec
2. Fetches the ServiceAccount from the Kubernetes API
3. Checks for the api.vultr.com/role annotation
4. If present, mutates the pod to inject:
   • Environment Variables:
     • AWS_ROLE_ARN: The IAM role ARN from the annotation
     • AWS_WEB_IDENTITY_TOKEN_FILE: Path to the projected token
     • AWS_STS_REGIONAL_ENDPOINTS: Set to "regional"
   • Volume: A projected ServiceAccount token volume with audience "vultr"
   • Volume Mounts: Mounts the token at /var/run/secrets/vultr.com/serviceaccount

Prerequisites

• Kubernetes 1.20+ (for projected service account tokens)
• kubectl configured to access your cluster
  • Deploy a VKE cluster and do export KUBECONFIG=~/Downloads/vke-64c243de-eb0b-4084-93ae-6c386bef8978.yaml
• OpenSSL (for certificate generation)
• Go 1.24+ (for building from source)

Quick Start

1. Build the Docker Image

docker build -t your-registry/irsa-webhook:latest .
docker push your-registry/irsa-webhook:latest

Update deploy.yaml with your image location.

2. Generate TLS Certificates

The webhook requires TLS certificates to communicate with the Kubernetes API server:

chmod +x generate-certs.sh
./generate-certs.sh

This script will:

• Generate a self-signed CA and certificate
• Create a Kubernetes secret with the certificates
• Update the MutatingWebhookConfiguration with the CA bundle

3. Deploy the Webhook

kubectl apply -f deploy.yaml

This creates:

• Namespace: irsa-system
• ServiceAccount with RBAC permissions
• Deployment with 2 replicas
• Service
• MutatingWebhookConfiguration

4. Verify Deployment

kubectl get pods -n irsa-system
kubectl logs -n irsa-system -l app=irsa-webhook

Usage

Annotate ServiceAccount

To enable IRSA for a ServiceAccount, add the api.vultr.com/role annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: default
  annotations:
    api.vultr.com/role: "775a6be6-45cd-4f19-94f5-6e4f96f093ec"

Deploy a Pod

Any pod using this ServiceAccount will automatically receive the AWS configuration:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  namespace: default
spec:
  serviceAccountName: my-app
  containers:
  - name: app
    image: amazon/aws-cli:latest
    command: ["sleep", "3600"]

Verify Injection

Check that the pod has the injected configuration:

# Check environment variables
kubectl exec my-app -- env | grep AWS

# Check volume mount
kubectl exec my-app -- ls -la /var/run/secrets/vultr.com/serviceaccount

# Test AWS credentials
kubectl exec my-app -- aws sts get-caller-identity

Configuration

Environment Variables

The webhook supports the following environment variables:

• TLS_CERT_PATH: Path to TLS certificate (default: /etc/webhook/certs/tls.crt)
• TLS_KEY_PATH: Path to TLS private key (default: /etc/webhook/certs/tls.key)
• PORT: HTTPS port to listen on (default: 8443)

Webhook Configuration

Edit the MutatingWebhookConfiguration in deploy.yaml:

• failurePolicy: Set to Fail for production to block pods if webhook is unavailable
• timeoutSeconds: Adjust timeout based on cluster performance
• namespaceSelector: Control which namespaces are affected

Security Considerations

1. Least Privilege: The webhook ServiceAccount only has permissions to read ServiceAccounts
2. TLS: All communication is encrypted using TLS 1.2+
3. Non-root: Container runs as non-root user (65532)
4. Read-only filesystem: Container has read-only root filesystem
5. No privilege escalation: Security context prevents privilege escalation

Troubleshooting

Webhook Not Mutating Pods

1. Check webhook logs:

   kubectl logs -n irsa-system -l app=irsa-webhook
   

2. Verify MutatingWebhookConfiguration:

   kubectl get mutatingwebhookconfiguration irsa-webhook -o yaml
   

3. Check if ServiceAccount has the annotation:

   kubectl get sa <service-account-name> -o yaml
   

Certificate Issues

If you see TLS errors, regenerate certificates:

./generate-certs.sh
kubectl rollout restart deployment -n irsa-system irsa-webhook

RBAC Permissions

If webhook can't fetch ServiceAccounts, verify RBAC:

kubectl auth can-i get serviceaccounts --as=system:serviceaccount:irsa-system:irsa-webhook --all-namespaces

Development

Local Testing

You can test the webhook logic locally:

package main

import (
    "testing"
    corev1 "k8s.io/api/core/v1"
)

func TestGeneratePatches(t *testing.T) {
    ws := &WebhookServer{}
    pod := &corev1.Pod{
        Spec: corev1.PodSpec{
            Containers: []corev1.Container{
                {Name: "test"},
            },
        },
    }

    patches, err := ws.generatePatches(pod, "arn:aws:iam::123456789012:role/test")
    if err != nil {
        t.Fatalf("Failed to generate patches: %v", err)
    }

    if len(patches) == 0 {
        t.Error("Expected patches to be generated")
    }
}

Building from Source

go mod download
go build -o webhook main.go

Architecture

┌─────────────┐
│ Kubernetes  │
│ API Server  │
└──────┬──────┘
       │
       │ AdmissionReview Request
       │
       ▼
┌─────────────────┐
│ IRSA Webhook    │
│                 │
│ 1. Parse Pod    │
│ 2. Get SA       │◄────┐
│ 3. Check Anno   │     │
│ 4. Gen Patches  │     │
└─────────────────┘     │
                        │
                  ┌─────┴──────┐
                  │ K8s API    │
                  │ (Get SA)   │
                  └────────────┘

Complete Flow

• Register your cluster's JWKS as an OIDC Issuer at https://api.vultr.com/v2/oidc/issuer
  • For VKE clusters:

  {
    "source": "vke",
    "source_id": "a070d34b-8380-441a-8fb4-d5a9c4001226" #This is the id of your VKE cluster
  }
  

  • For NON VKE clusters:

  {
    "source": "external",
    "uri": "https://64c243de-eb0b-4084-93ae-6c386bef8978.vultr-k8s.com:6443/openid/v1/jwks",
    "kid": "Sf4VzjgTmm_pW91u5qZypZWwiac9_boRFPC5vEmuhCQ",
    "kty": "RSA",
    "n": "3nhZuoDdSSr6OvdnxfOiJKZoC3kcnuEqbJyxXx0ULZLld3rxOmY8w1cuVjNIOaQsZZzQ6qeR7Z315L-Cdi19SLJRcdPf4d0Nezj9pmE_C0VjyNa8w0ZeF23xgiSnE4-ZamLdPtmxWXGhyyBSc_3CRBo-yFdAYJrsmXT1jjm_DOFpI3ZnKqeK7zmG9pRK-OaXfIXw_PEAZ3scflUkv1tE_j21YnFYd8BSM_He_V4Wx3MRFEBqr9-NbVegsEaQsZU63G_BCxEQXHXXM1YJ9ubE29jvMUrSHNFrgLrAjQhXrwu-PpEU1ROwbG4G0FaWkxEzC2K2_gqVC-Q4g-eEYS73UQ",
    "e": "AQAB",
    "alg": "RS256",
    "use": "sig"
  }
  

• From this point you will be able to auth to the vultr API from inside your kubernetes cluster using the standard - See the file in this repo test-oidc-issuer.yaml
• Deploy this irsa-webhook to your cluster
• Pod->STS
  • Now when when a pod is owned by a serviceAccount with the annotation api.vultr.com/role, the pod will send a token issued by the cluster to the Vultr sts endpoint.
• STS->Pod
  • Vultr's STS endpoint will respond with tokens issued by Vultr that are injected into the pod for the application running in the pod to consume

The Full Flow with Both Tokens

┌──────────────────────────────────────────────────────────────────────────────┐
│ YOUR CLUSTER                                                                 │
│                                                                              │
│ Kubernetes API Server (configured with your issuer)                          │
│ ├─ Generates TOKEN #1 (ServiceAccount JWT)                                   │
│ │  Signed with: cluster's private key                                        │
│ │  Claims:                                                                   │
│ │    iss: "https://api.vultr.com/v2/oidc"                                    │
│ │    aud: "vultr"                                                            │
│ │    sub: "system:serviceaccount:default:test-sa"                            │
│ └─ Mounts TOKEN #1 in pod at:                                                │
│    /var/run/secrets/kubernetes.io/serviceaccount/token                       │
│                                                                              │
│ ┌────────────────────────────────────────────────────────────────────────┐   │
│ │ Pod: my-app                                                            │   │
│ │                                                                        │   │
│ │ 1. Application starts                                                  │   │
│ │ 2. SDK reads TOKEN #1 from file                                        │   │
│ │ 3. SDK calls Vultr STS with TOKEN #1                                   │   │
│ └────────────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────────┘

                             │
TOKEN #1 (K8s JWT) sent to Vultr platform ────────────────────────────────────┘
                             ↓

┌──────────────────────────────────────────────────────────────────────────────┐
│ VULTR PLATFORM (api.vultr.com)                                               │
│                                                                              │
│ STS Service                                                                  │
│ ├─ Receives TOKEN #1 from pod                                                │
│ ├─ Validates TOKEN #1:                                                       │
│ │  └─ Fetches public key from /v2/oidc/jwks                                  │
│ │  └─ Verifies signature                                                     │
│ │  └─ Checks issuer, audience, expiration                                    │
│ │  └─ Checks role trust policy                                               │
│ ├─ Generates TOKEN #2 (Temporary Credentials)                                │
│ │  └─ AccessKeyId: VKAEXAMPLE123ABC                                          │
│ │  └─ SecretAccessKey: secretKEY789XYZ                                       │
│ │  └─ SessionToken: sessionTOKEN456DEF                                       │
│ └─ Returns TOKEN #2 to pod                                                   │
└──────────────────────────────────────────────────────────────────────────────┘

                             │
TOKEN #2 (Temporary credentials) sent back to pod ────────────────────────────┘
                             ↓

┌──────────────────────────────────────────────────────────────────────────────┐
│ YOUR CLUSTER                                                                 │
│                                                                              │
│ ┌────────────────────────────────────────────────────────────────────────┐   │
│ │ Pod: my-app                                                            │   │
│ │                                                                        │   │
│ │ 4. SDK receives TOKEN #2 (credentials)                                 │   │
│ │ 5. SDK caches TOKEN #2                                                 │   │
│ │ 6. SDK uses TOKEN #2 for all API calls:                                │   │
│ │    - List buckets                                                      │   │
│ │    - Upload objects                                                    │   │
│ │    - etc.                                                              │   │
│ └────────────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────────┘

                             │
All API calls use TOKEN #2 (credentials) ────────────────────────────────────┘
                             ↓

┌──────────────────────────────────────────────────────────────────────────────┐
│ VULTR PLATFORM APIs (api.vultr.com/v2/*)                                     │
│                                                                              │
│ Object Storage API, Compute API, etc.                                        │
│ ├─ Receives request with TOKEN #2 (SessionToken)                             │
│ ├─ Validates TOKEN #2 against session database                               │
│ ├─ Checks permissions from role                                              │
│ └─ Executes API operation                                                    │
└──────────────────────────────────────────────────────────────────────────────┘