My Posts

Part 2: Initializing Vault, Enabling IAM Authentication, and Securing Access

Tue May 27 11:31:42 2025

Part 2: Initializing Vault, Enabling IAM Authentication, and Securing Access This is part 2 of the Vault configuration where I will cover how we can enable IAM authentication for simplicity and because for example if we heavily using AWS services. I will cover: - How to create & configure AWS roles across organization for Vault - Create example role and provide test policy for it - Configure external-secrets to fetch the secret from the Vault As in the part 1 we succesfully deployed our EKS cluster and initialized our Vault cluster, not it's time to confiugre access to the Vault server. Let's imagine that you're using AWS provider as a main cloud provider and do not reinvent the wheel we need to stick with IAM roles we don't want to manage Vault tokens and we will use AWS roles instead for authentication and autherization it's 100% achiveable but before starting we need to create AWS roles across AWS acounts where the Vault will be used. In the part one we deployed our Vault server as a helm chart and we provided to the vault base role using IRSA approach, for example there was vault AWS acount, now we want access vault from the Dev account Firstly we need to create this role in Dev account. Again as it was previous I will use terragrunt / tofu for this purposes. create the policy first


include "root" {
  path = find_in_parent_folders()
}

include "aws" {
  path = find_in_parent_folders("aws.hcl")
}

terraform {
  source = "tfr:///terraform-aws-modules/iam/aws//modules/iam-policy?version=${local.version}"
}

locals {
  config  = jsondecode(read_tfvars_file(find_in_parent_folders("config.tfvars")))
  version = read_terragrunt_config(find_in_parent_folders("versions.hcl")).locals.terraform.iam_policy
}

inputs = {
  name = "vault-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "ec2:DescribeInstances",
          "iam:GetInstanceProfile",
          "iam:GetUser",
          "iam:GetRole"
        ],
        Resource = "*"
      }
    ]
  })
}

An the role itself which allows to assume from IAM role from account 11111111111


include "root" {
  path = find_in_parent_folders()
}

include "aws" {
  path = find_in_parent_folders("aws.hcl")
}

dependency "management_vault_role" {
  config_path = "${get_path_to_repo_root()}/aws/infrastructure/services/global/iam/role/vault/role"
}

dependency "policy" {
  config_path = "${get_original_terragrunt_dir()}/../policy"
}

terraform {
  source = "tfr:///terraform-aws-modules/iam/aws//modules/iam-assumable-role?version=${local.version}"
}

locals {
  version = read_terragrunt_config(find_in_parent_folders("versions.hcl")).locals.terraform.iam_oidc
  config  = jsondecode(read_tfvars_file(find_in_parent_folders("config.tfvars")))
}

inputs = {
  create_role       = true
  role_name         = "vault"
  role_requires_mfa = false
  trusted_role_arns = [
    #Dev account Vault role
    dependency.management_vault_role.outputs.iam_role_arn
  ]
  custom_role_policy_arns = [
    dependency.policy.outputs.arn
  ]
}

After we have created `Vault` role from the Dev account, we need somehow tell the Vault that it can use this role to check all the roles from the account 22222222222 and for this purpose again I will create custom terraform module. It's pretty straight forward but I will put the code snippet from this module.


#main.tf
resource "vault_auth_backend" "aws" {
  type = "aws"
}

resource "vault_aws_auth_backend_sts_role" "accounts" {
  for_each = var.aws_accounts

  backend              = vault_auth_backend.aws.path
  account_id           = each.value.account_id
  sts_role             = each.value.sts_role
}

resource "vault_policy" "example" {
  for_each = var.policies

  name = each.key
  policy = each.value.policy
}

resource "vault_aws_auth_backend_role" "aws" {
  for_each = var.roles
  role    = each.key

  auth_type = "iam"
  backend = vault_auth_backend.aws.path
  bound_iam_principal_arns = each.value.role_arn

  token_policies = each.value.policies
  token_max_ttl =  each.value.max_ttl
}

resource "vault_mount" "kvv2" {
  path        = "secret"
  type        = "kv"
  description = "Key-Value secret engine v2 at secret/"

  options = {
    version = "2"
  }
}

#variables.tf
variable "policies" {
  description = "List of policies to attach to the Vault role"
  type        = map(object({
    policy = string
  }))
  default = {}
}

variable "roles" {
  description = "List of roles to attach which allows access Vault"
  type        = map(object({
    role_arn = list(string)
    policies = list(string)
    max_ttl  = number
  }))
  default = {}
}

variable "aws_accounts" {
  description = "List of AWS accounts to serve as a trust relationship for the Vault server"
  type        = map(object({
    account_id = string
    sts_role   = string
  }))
  default     = {}
}

#versions.tf
terraform {
  required_providers {
    vault = {
      source = "hashicorp/vault"
      version = "~> 4.8.0"
    }
  }
  required_version = ">= 1.3.0"
}

After we added this module we can use it


include "root" {
  path = find_in_parent_folders()
}

include "vault" {
  path = find_in_parent_folders("vault.hcl")
}

include "aws" {
  path = find_in_parent_folders("aws.hcl")
}

dependency "dev-external-secret-role" {
  config_path = "${get_path_to_repo_root()}/aws/workloads/dev/global/iam/role/external-secrets/role"
}

dependency "dev-vault-role" {
  config_path = "${get_path_to_repo_root()}/aws/workloads/dev/global/iam/role/vault/role"
}

terraform {
  source = "${get_path_to_repo_root()}//modules/vault"
}

locals {
  config = jsondecode(read_tfvars_file(find_in_parent_folders("config.tfvars")))
}

inputs = {
  aws_accounts = {
    dev = {
      account_id = local.config.accounts[index(local.config.accounts.*.name, "dev")].id
      sts_role   = dependency.dev-vault-role.outputs.iam_role_arn
    }
    stage = {
      account_id = local.config.accounts[index(local.config.accounts.*.name, "stage")].id
      sts_role   = dependency.stage-vault-role.outputs.iam_role_arn
    }
    prod = {
      account_id = local.config.accounts[index(local.config.accounts.*.name, "prod")].id
      sts_role   = dependency.prod-vault-role.outputs.iam_role_arn
    }
  }

  policies = {
    dev-external-secrets = {
      policy = <<-EOF
        path "/secret/data/dev/*" {
          capabilities = ["read", "list"]
        }
      EOF
    }
  }

  roles = {
    dev-external-secrets = {
      role_arn = [dependency.dev-external-secret-role.outputs.iam_role_arn]
      policies = ["dev-external-secrets"]
      max_ttl  = 120
    }
  }
}

After we succesfully configured our vault we can test it, code snippet below contains yaml manifest where we can test that external-secret can communicate with Vault and fetch the secrets, I'm assuming that you already deployed external-secrets operator with CRD's.


apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
spec:
  provider:
    vault:
      server: "https://vault.services.internal.com"
      path: "secret"
      version: "v2"
      auth:
        iam:
          path: aws
          region: us-east-1
          vaultRole: dev-external-secrets
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-example
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-store
    kind: SecretStore
  target:
    name: example-sync
  data:
  - secretKey: example-key
    remoteRef:
      key: dev/k8s/test/username

After we applied our manifest we can check it that everything is wokring fine.


kubectl describe secretstore,externalsecret
Name:         vault-store
Namespace:    default
Labels:       
Annotations:  
API Version:  external-secrets.io/v1beta1
Kind:         SecretStore
Metadata:
  Creation Timestamp:  2025-05-23T10:53:46Z
  Generation:          1
  Resource Version:    87589679
  UID:                 ab6c4fce-3c08-46e9-aa90-1dff1583d2b2
Spec:
  Provider:
    Vault:
      Auth:
        Iam:
          Path:        aws
          Region:      us-east-1
          Vault Role:  dev-external-secrets
      Path:            secret
      Server:          https://vault.services.internal.com
      Version:         v2
Status:
  Capabilities:  ReadWrite
  Conditions:
    Last Transition Time:  2025-05-23T10:53:47Z
    Message:               store validated
    Reason:                Valid
    Status:                True
    Type:                  Ready
Events:
  Type    Reason  Age              From          Message
  ----    ------  ----             ----          -------
  Normal  Valid   2s (x2 over 2s)  secret-store  store validated
  Normal  Valid   2s (x2 over 2s)  secret-store  store validated


Name:         vault-example
Namespace:    default
Labels:       
Annotations:  
API Version:  external-secrets.io/v1beta1
Kind:         ExternalSecret
Metadata:
  Creation Timestamp:  2025-05-23T10:53:47Z
  Generation:          1
  Resource Version:    87589687
  UID:                 c8e7ff26-ca33-4b0d-a00a-f5600eeafab7
Spec:
  Data:
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  dev/k8s/test/username
      Metadata Policy:      None
    Secret Key:             example-key
  Refresh Interval:         15s
  Secret Store Ref:
    Kind:  SecretStore
    Name:  vault-store
  Target:
    Creation Policy:  Owner
    Deletion Policy:  Retain
    Name:             example-sync
Status:
  Binding:
    Name:  example-sync
  Conditions:
    Last Transition Time:   2025-05-23T10:53:47Z
    Message:                secret synced
    Reason:                 SecretSynced
    Status:                 True
    Type:                   Ready
  Refresh Time:             2025-05-23T10:53:47Z
  Synced Resource Version:  1-af61a938cce70a6fbb5c7dae966c7503
Events:
  Type     Reason        Age              From              Message
  ----     ------        ----             ----              -------
  Warning  UpdateFailed  2s (x5 over 2s)  external-secrets  error retrieving secret at .data[0], key: dev/k8s/test/username, err: SecretStore "vault-store" is not ready
  Warning  UpdateFailed  2s (x5 over 2s)  external-secrets  error retrieving secret at .data[0], key: dev/k8s/test/username, err: SecretStore "vault-store" is not ready
  Normal   Created       2s               external-secrets  Created Secret
  Warning  UpdateFailed  2s               external-secrets  secrets "example-sync" already exists

As you can see secret was created and your application can consumed it, I undersand security concern here, better to directly support into your application with Vault communication but as a starter point it's ready to go.

Comments

To leave a comment please login via github