This document describes how to verify a Cloud Deploy deployment.

You can configure Cloud Deploy to verify that an
application that you have deployed to any target is working correctly.
Verification is done using your own testing image or images, and you configure
Cloud Deploy to run those tests after deployment
finishes.

## How does deployment verification work?

1. You [configure](https://docs.cloud.google.com/deploy/docs/verify-deployment#configure_pipeline) one or more targets in your delivery
   pipeline for deployment verification by defining [tasks](https://docs.cloud.google.com/deploy/docs/tasks)
   to run.

2. After an application is deployed, Cloud Deploy runs
   the verify tasks in the Cloud Deploy execution environment.

   Success or failure of the tests executed indicates success or failure of the
   verification:
   - Verification success is determined by the exit code generated by the container.

     `0` indicates success. A non-zero exit code indicates failure. To generate
     the expected verification result, make sure the container exits with the
     appropriate exit code. If more than one container is executed as part of
     verification, they must all succeed for verification to succeed.
   - If the verification fails, the rollout also fails.

   - If a deployment fails during verification, you can see that by inspecting
     the rollout:

     ![Details in Google Cloud console for rollout, including verification status](https://docs.cloud.google.com/static/deploy/images/rollouts_verification_failure1.png)
3. You can [ignore](https://docs.cloud.google.com/deploy/docs/deployment-strategies/manage-rollout#ignore_job) or [retry](https://docs.cloud.google.com/deploy/docs/verify-deployment#retry) a failed verification.

   You can also [terminate](https://docs.cloud.google.com/deploy/docs/deployment-strategies/manage-rollout#terminate_job_run)
   an in-progress verify job.

### Components used for verification

The [rollout](https://docs.cloud.google.com/deploy/docs/architecture#resources) resource includes the
following objects, which support deployment verification:

- Phase

  The collection of operations (jobs) in a rollout that are logically grouped
  together, for example a deploy or a deploy and verify.
- Job

  The specific operation to be performed on a rollout, such as deploy or verify.
- Job run

  A child of the rollout resource, the job run is an instance of a job, for
  example an attempt to deploy.

For more about Cloud Deploy resources, see [Cloud Deploy
service architecture](https://docs.cloud.google.com/deploy/docs/architecture)

## Configure Cloud Deploy for deployment verification

Enabling deployment verification for a Cloud Deploy target consists
of adding a `verify` stanza to a given target (or targets) in a delivery
pipeline progression, as shown in this example:

    apiVersion: deploy.cloud.google.com/v1
    kind: DeliveryPipeline
    metadata:
     name: my-demo-app
    description: main application pipeline
    serialPipeline:
     stages:
     - targetId: dev
       profiles: []
       strategy:
         standard:
           verify:
             tasks:
             - type: container
               image: "VERIFY_IMAGE"
               command: [COMMANDS_TO_RUN]
               args: [LIST_OF_ARGS]
               env: {VERIFY_TASK_ENV_MAP}

In this YAML:

- <var translate="no">VERIFY_IMAGE</var>

  Is the name of the image you want to run for the verify job. For example,
  `us-central1-docker.pkg.dev/gcp-project-id-12345/my-repository/my-app:v1.2`
  for an Artifact Registry image.
- <var translate="no">COMMANDS_TO_RUN</var>

  Is a list of entrypoints to run on that container. `"/bin/sh"` is a typical
  command to specify here, to invoke a shell.
- <var translate="no">LIST_OF_ARGS</var>

  Is a list of arguments to provide to the command. This is a comma-separated
  list, with each argument in quotation marks. If your <var translate="no">COMMAND_TO_RUN</var>
  is `"/bin/sh"`, then one of the arguments here would be `"-c"`, and another
  argument would be the entire command you want to run in the shell you're
  invoking.

  Here's an example:

      command: ["/bin/sh"]
      args: ["-c", `echo "This command ran!"`]

- <var translate="no">VERIFY_TASK_ENV_MAP</var>

  Is a map of [environment variables](https://docs.cloud.google.com/deploy/docs/verify-deployment#available_environment_variables) passed into the container, in the format
  `KEY:VAL`.

The verify operation is executed within its own [execution environment](https://docs.cloud.google.com/deploy/docs/execution-environment). This execution environment can
be configured for `VERIFY` the same as it can be for `RENDER` and `DEPLOY`.

## Run verification on the application cluster

By default, deployment verification runs in the Cloud Deploy
[execution environment](https://docs.cloud.google.com/deploy/docs/execution-environment). You can also
configure Skaffold to run verification on the same cluster where your
application is running.

To run your verification containers on the cluster, you must configure the verification
containers under the `verify` stanza in your `skaffold.yaml`. For each container
defined, you must also set `executionMode.kubernetesCluster`.

    verify:
    - name:
      container:
        name:
        image:
        command:
        args:
      executionMode:
        kubernetesCluster:

The following is an example verify stanza that includes `executionMode` to invoke the verification container on the application cluster:

    verify:
    - name: integration-test-container
      container:
        name: integration-test-container
        image: integration-test-container
      executionMode:
        kubernetesCluster: {}

In addition, you must set the `verify` stanza to `true` inside your delivery pipeline configuration.

The following is an example delivery pipeline definition where the `dev` target
has verification enabled.

    apiVersion: deploy.cloud.google.com/v1
    kind: DeliveryPipeline
    metadata:
     name: my-demo-app
    description: main application pipeline
    serialPipeline:
     stages:
     - targetId: dev
       profiles: []
       strategy:
         standard:
           verify: true

This ability is available for deployments to GKE only, not for
Cloud Run. Deployments to Cloud Run can only
run verification in the Cloud Deploy execution environment.

For [parallel deployments](https://docs.cloud.google.com/deploy/docs/parallel), verification is
configured on the [multi-target](https://docs.cloud.google.com/deploy/docs/terminology#multi-target), and the
verification containers run on each
[child target](https://docs.cloud.google.com/deploy/docs/terminology#child_target).

You can also include the `jobManifestPath` and `overrides` properties,
under `kubernetesCluster`, to point to a manifest for your verify container, and
any values you want to override. (`overrides` takes Kubernetes inline JSON with
the values you want to replace.) [Learn more](https://skaffold.dev/docs/verify/).

The `executionMode` stanza is optional, and if you omit it, Skaffold runs the
verification containers in the Cloud Deploy execution environment.

> [!NOTE]
> **Note:** If you configure verification using `tasks`, as shown in the [section above](https://docs.cloud.google.com/deploy/docs/verify-deployment#configure_pipeline), those verification containers run only in the Cloud Deploy execution environment. To run verification on the cluster, you must define the `verify` stanza in `skaffold.yaml`.

## Retry the verification

When a verify job fails, you can
[retry](https://docs.cloud.google.com/deploy/docs/deployment-strategies/manage-rollout#retry_failed_job) the
verification, creating a new job run:

    gcloud deploy rollouts retry-job ROLLOUT_NAME \
                 --job-id=JOB_ID \
                 --phase-id=PHASE_ID \
                 --delivery-pipeline=PIPELINE_NAME \
                 --release=RELEASE_NAME \
                 --region=REGION

Retrying the verification changes the state of the rollout from `FAILED` to
`IN_PROGRESS`.

You can only retry a verification for a rollout whose verify job failed.

## Available environment variables

Cloud Deploy also provides and populates the following environment
variables in the [execution environment](https://docs.cloud.google.com/deploy/docs/execution-environment).
You can use these environment variables as part of your
[deploy hook](https://docs.cloud.google.com/deploy/docs/hooks), [verify job](https://docs.cloud.google.com/deploy/docs/verify-deployment),
or [custom target](https://docs.cloud.google.com/deploy/docs/custom-targets) render or deploy.

- `ANTHOS_MEMBERSHIP`

  For targets of type `ANTHOS`, the fully specified resource name of the Anthos
  membership.
- `CLOUD_RUN_LOCATION`

  For targets of type `RUN`, the region the Cloud Run service is
  deployed in.
- `CLOUD_RUN_PROJECT`

  For targets of type `RUN`, the project in which the Cloud Run
  service was created.
- `CLOUD_RUN_SERVICE`

  For targets of type `RUN`, the name of the Cloud Run service
  deployed.
- `CLOUD_RUN_SERVICE_URLS`

  For targets of type `RUN`, the URL or URLs (comma-separated list) that end
  users will use to access your service. You can find these in the
  Cloud Run service details for your service, in the
  Google Cloud console. The URLs are generated by Cloud Run after your Cloud Run Service or Services have been successfully deployed. Therefore this environment variable is only available in postdeploy hooks and
  [verify jobs](https://docs.cloud.google.com/deploy/docs/verify-deployment).
- `CLOUD_RUN_REVISION`

  For targets of type `RUN`, the specific revision of the Cloud Run
  service.
- `GKE_CLUSTER`

  For targets of type `GKE`, the fully specified resource name of the
  Google Kubernetes Engine cluster, for example `projects/p/locations/us-central1/clusters/dev`.
- `TARGET_TYPE`

  The specific runtime type of the target. Either `GKE`, `ANTHOS`, or `RUN`.
  For custom targets, this won't be set.
- `CLOUD_DEPLOY_LOCATION`

  The region containing the Cloud Deploy resources.
- `CLOUD_DEPLOY_DELIVERY_PIPELINE`

  The ID of the delivery pipeline.
- `CLOUD_DEPLOY_TARGET`

  The ID of the target.
- `CLOUD_DEPLOY_PROJECT`

  The Google Cloud project number for the project containing the
  Cloud Deploy resources.
- `CLOUD_DEPLOY_PROJECT_ID`

  The Google Cloud project ID for the project.
- `CLOUD_DEPLOY_RELEASE`

  The ID of the release in which the hooks will run.
- `CLOUD_DEPLOY_ROLLOUT`

  The ID of the rollout that contains the jobs for the hooks.
- `CLOUD_DEPLOY_JOB_RUN`

  The ID of the [job run](https://docs.cloud.google.com/deploy/docs/deployment-strategies/manage-rollout#jobs_and_job_runs) that represents the current execution of the
  job.
- `CLOUD_DEPLOY_PHASE`

  The [phase](https://docs.cloud.google.com/deploy/docs/deployment-strategies/manage-rollout#phases) in the rollout that contains the job for the deploy hook, verify job, or custom render or deploy.

## What's next

- Learn more about [tasks](https://docs.cloud.google.com/deploy/docs/tasks)
- Try the [quickstart: Verify your deployment](https://docs.cloud.google.com/deploy/docs/deploy-app-verification)