Improve argocd app sync --dry-run for Applications with New Resources

[jsolana]

Summary

The argocd app sync --dry-run command does not work as expected when new resources are added to an existing application. These new resources are not detected, which means any issues with the added resources are not surfaced during the dry-run process.

Motivation

A robust --dry-run feature is critical for validating changes before they are applied, especially during pull request reviews. If new resources being added to an application are not properly validated during a dry-run, potential problems (e.g., misconfigurations, policy violations) could go undetected until deployment, increasing the risk of production issues.

Example Use Case

A developer extends an existing application by adding new resources. They perform a dry-run to ensure that the changes are correct and compliant with policies. However, since the dry-run does not detect these new resources, issues such as invalid configurations or policy violations are only discovered during deployment, rather than earlier in the pull request process.

How to reproduce it (using this repo):

kind create cluster
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Installing kyverno an policies from  https://github.com/jsolana/argocd-applications
kubectl apply -f kyverno-app.yml
kubectl apply -f kyverno-policies-app.yml

# installing dummy app
kubectl apply -f busybox-echo.yml

# login
kubectl port-forward svc/argocd-server -n argocd 8080:443
argocd login localhost:8080 --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo) --insecure

Updating the already existent deployment adding latest tag:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-dev-echo
  labels:
    tier: "3"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox-dev-echo
  template:
    metadata:
      labels:
        app: busybox-dev-echo
        tier: "3"
    spec:
      containers:
      - name: busybox
        image: busybox:latest
        command: ["/bin/sh", "-c"]
        args: ["while true; do echo Hello Dev; sleep 5; done"]
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"

And running sync --dry-run command, the result is expected, failing cause disallow-latest-tag validation fails:

argocd app sync busybox-echo --dry-run --timeout 60 --retry-limit 1 --local <path>
...

Message:            ComparisonError: Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error running server side apply in dryrun mode for resource Deployment/busybox-dev-echo: admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/busybox-echo/busybox-dev-echo was blocked due to the following policies 

disallow-latest-tag:
  autogen-validate-image-tag: 'validation failure: validation error: Using a mutable
    image tag e.g. ''latest'' is not allowed. rule autogen-validate-image-tag failed
    at path /image/' (retried 1 times).

If we create a new resource with the latest tag, eg, a StatefulSet:

# StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-statefulset
  namespace: busybox-echo
  labels:
    app: helloworld
spec:
  serviceName: "nginx-service"
  replicas: 2
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Running the sync --dry-run commands ends succesfully:

Message:            successfully synced (no more tasks)

GROUP  KIND         NAMESPACE     NAME               STATUS     HEALTH   HOOK  MESSAGE
apps   Deployment   busybox-echo  busybox-dev-echo   Synced     Healthy        deployment.apps/busybox-dev-echo configured (dry run)
apps   StatefulSet  busybox-echo  nginx-statefulset  Succeeded  Synced         statefulset.apps/nginx-statefulset created (dry run)

But running kubectl apply -f sts.yml --dry-run=server trigger the error expected:

Error from server: error when creating "sts.yml": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource StatefulSet/busybox-echo/nginx-statefulset was blocked due to the following policies 

disallow-latest-tag:
  autogen-validate-image-tag: 'validation failure: validation error: Using a mutable
    image tag e.g. ''latest'' is not allowed. rule autogen-validate-image-tag failed
    at path /image/'
validate-labels:
  autogen-require-tier-label: 'validation error: Label `tier` with values 1, 2, 3
    or 4 must be set on all pods. rule autogen-require-tier-label failed at path /spec/template/metadata/labels/tier/'

Proposal

To improve the argocd app sync --dry-run command, it should handle new resources in the same way it handles modifications to existing resources:

Ensure that new resources are detected and validated during a dry-run.
Internally perform a server-side diff and server-side apply for the new resources, invoking admission webhooks (e.g., Kyverno) to validate the changes.

Provide detailed feedback in the dry-run output, highlighting any issues with the new resources (e.g., validation errors, policy violations).
This behavior would align the dry-run functionality for new resources with its behavior for modifications, providing consistent and comprehensive validation. It would also help developers detect and address issues earlier in the development workflow, such as during pull request reviews.

@leoluz ServerSide Diff is an expensive operation as it needs to reach KubeAPI to calculate diffs. The decision to avoid that logic on resource creation was based on this limitation.

Maybe make sense to create a new syncOption or a specific value for ServerSideDiff to allow configure creation coverage.

References

• More info here

[jsolana]

Maybe duplicated argoproj/gitops-engine#631