How to configure Argocd-vault-plugin as Sidecar by Helm installation

1. Introduction

Many find it challenging to implement secret protection with GitOps using ArgoCD and the argocd-vault-plugin. This blog post demonstrates how to install the plugin with ArgoCD using the Helm chart.

2. What is argocd-vault-plugin (AVP)?

Source AVP is a plugin that fetches secrets from a secret store (such as Vault) on the fly when ArgoCD syncs applications.

3. Common Challenges When Installing AVP

• Confusing documentation: Multiple implementation methods exist, but they are not clearly explained. (I prefer not to use custom Docker images.)

• Lack of online resources: Most guides cover older installation methods, not sidecar setups using Helm.

• Configuration complexity: Understanding sidecar volume mounts and ConfigManagementPlugin (CMP) requires combining multiple parts, which can be overwhelming.

• Secret ingestion: Overcoming secret injection challenges is the final hurdle.

4. Step Summary

1. Predefine Vault secret: ArgoCD requires existing Vault installation and a secret with connection details.

2. Add plugin configuration: Define how the plugin operates.

3. Edit ArgoCD deployment: Configure the plugin sidecar.

4. Create an Application: Test the installation.

5. Installation Steps

5.1 Predefine Vault Secret

The secret must be created manually because ArgoCD needs it to fetch other secrets. You can store it in Git using SOPS or Sealed Secrets.

apiVersion: v1
kind: Secret
metadata:
  name: argocd-vault-plugin-credentials
  namespace: argocd
stringData:
  VAULT_ADDR: "http://...:8200"
  VAULT_TOKEN: ...
  AVP_AUTH_TYPE: token
  AVP_TYPE: vault

• You can place the secret in any namespace (explained below).

• Full backend storage configuration: Source

5.2 Add Plugin Configuration

This part may seem trivial, but it is crucial. Pay close attention here. Since this section contains no secrets, you can store it in Git. I focus on plain YAML and Helm plugins, skipping Kustomize and Jsonnet.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
  namespace: argocd
data:
  avp.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find $ARGOCD_ENV_CHART_PATH -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "-s"
          - "argocd:$ARGOCD_ENV_VAULT_SECRET"
          - "$ARGOCD_ENV_CHART_PATH"
      lockRepo: false
  avp-helm.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find $ARGOCD_ENV_CHART_PATH -name 'Chart.yaml' && find $ARGOCD_ENV_CHART_PATH -name 'values.yaml'"
      generate:
        command:
          - sh
          - "-c"
          - |
            helm template  $ARGOCD_APP_NAME $ARGOCD_ENV_HELM_ARGS -n $ARGOCD_APP_NAMESPACE --include-crds . | argocd-vault-plugin generate -s argocd:$ARGOCD_ENV_VAULT_SECRET - 
      lockRepo: false

Key Sections:

• Source: CMP config

• discover: Determines which plugin to use (YAML or Helm) by running simple commands.

• generate: Executes the plugin's actions. For Helm, helm template generates manifests, which argocd-vault-plugin generate processes.

• -s argocd:$ARGOCD_ENV_VAULT_SECRET specifies <namespace>:<vault-secret-name>.

• $ARGOCD_APP_* environment variables are predefined here.

• Custom environment variables, like $ARGOCD_ENV_VAULT_SECRET and $ARGOCD_ENV_HELM_ARGS, I will use this after.

5.3 Edit ArgoCD Deployment

Add the sidecar to the repo-server deployment by modifying the Helm chart. Ensure the ArgoCD version matches the plugin.

# https://artifacthub.io/packages/helm/argo/argo-cd
# Not very High-availability
redis-ha:
  enabled: false
crds:
  install: true
  # set:
  #   adminPassword: "NotVerySafePassword"
controller:
  replicas: 1
applicationSet:
  replicas: 1
configs:
  cm:
    timeout.reconciliation: 10s
server:
  replicas: 1
  autoscaling:
    enabled: false
  automountServiceAccountToken: true
repoServer:
  replicas: 1
  autoscaling:
    enabled: false
  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps
  initContainers:
  - name: download-tools
    image: alpine:3.8
    command: [sh, -c]

    # Don't forget to update this to whatever the stable release version is
    # Note the lack of the `v` prefix unlike the git tag
    env:
      - name: AVP_VERSION
        value: "1.18.1"
    args:
      - >-
        wget -O argocd-vault-plugin
        https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
        chmod +x argocd-vault-plugin &&
        mv argocd-vault-plugin /custom-tools/
    volumeMounts:
      - mountPath: /custom-tools
        name: custom-tools
  extraContainers:
    - name: avp-helm
      command: [/usr/local/bin/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.14.2
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumes:
        - name: cmp-plugin
          configMap:
            name: cmp-plugin
        - name: custom-tools
          configMap:
            name: custom-tools
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp
        # Register plugins into sidecar
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: avp-helm.yaml
          name: cmp-plugin
        # Important: Mount tools into $PATH
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin
    - name: avp
      command: [/usr/local/bin/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.14.2
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumes:
        - name: cmp-plugin
          configMap:
            name: cmp-plugin
        - name: custom-tools
          configMap:
            name: custom-tools
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp
        # Register plugins into sidecar
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: avp.yaml
          name: cmp-plugin
        # Important: Mount tools into $PATH
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

5.4 Create an Application

Here’s an example of a simple Helm chart using Vault secrets.

# $> helm create test
# $> vi test/template/test-configmap.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
  namespace: default
  annotations:
    avp.kubernetes.io/path: "secret/data/test"
    # <bucket>/data/<path> in Vault
    # bucket kv `secret`, path `test`
data:
  aaa: <secret1>
  bbb: <secret2>

Create an Application with spec.source.plugin type (other than helm/directory... type). If not, it causes conflict.

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: ./            # Helm chart path in git repo
    plugin:
      # name: argocd-vault-plugin-helm              # You don't need to specific plugin name, Argocd auto
                                                    # detect them by `discover` section !
      env:
        - name: CHART_PATH              # Not use.
          value: "."
        - name: VAULT_SECRET                        # Remember $ARGOCD_ENV_VAULT_SECRET ?
          value: "argocd-vault-plugin-credentials"  # Predefined secret name
        # Helm
        - name: HELM_ARGS 
          value: "-f values.yaml -f values-stg.yaml" # More flags with helm
    repoURL: https://[git-repo].git
    targetRevision: main
  syncPolicy:
    automated: {}

6. Q&A

Why not use External Secret Operator, SOPS, or Sealed Secrets?

• External Secret Operator only fetches secrets into Kubernetes secrets, not Helm values.

• SOPS/Sealed Secrets make secrets harder to access and edit.

How can I implement this with a local Helm chart?

Say you have a local helm chart with full template files, values files,...

.
├── Chart.yaml
├── templates
│   ├── _secret.tpl
│   ├── ...
└── values.yaml

... and your values.yaml

x:
  password: a   # this will be templated into Secret

Remember how argocd-vault-plugin works as I say above: (1)Template -> (2) Fetch secret from vault -> (3) Deploy. Then at step (2) it need to know Vault path to fetch secret, then at step (1) you need to fix some places.

# templates/_secret.tpl
...
data:
  token: {{ default "" .Values.x.password | quote }}
...
# Change to
...
stringData:
  token: {{ default "" .Values.x.password | quote }}
...

#----------------------------
# values.yaml
...
x:
  password: a
...
# Change to
...
x:
  password: <path:secret/data/test#secret1 | base64encode>
...
# List function
# [Source](https://argocd-vault-plugin.readthedocs.io/en/stable/howitworks/)

This solution you need to know which key-value pairs in values.yaml will be parsed to Secret, and you need to change secret template as well.

Is there a better way with Helm repositories?

I implement helm-controller Operator (k3s project (Source), not flux project). By this way, I can set vault place holder directly into "values.yaml file"

---
# API: [text](https://docs.k3s.io/helm)
# https://artifacthub.io/packages/helm/argo/argo-cd
# Define Helm chart source
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: argocd
  namespace: helmchart
spec:
  chart: argo-cd
  targetNamespace: argocd
  createNamespace: true
  version: 7.8.2
  repo: https://argoproj.github.io/argo-helm
---
# Define Helm chart values
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: argocd
  namespace: helmchart
spec:
  valuesContent: |-
    redis-ha:
      enabled: false
    crds:
      install: true
      set:
        adminPassword: <path:secret/data/test#secret1>

In this case, I put directly secret into values config, because this manifest go straight throught helm template command.

Other Tips

• You can download argo-vault-plugin binary to local and call command at discover,generate section at home to test them.

• For error handling, refer to this guide.