Cloud Tasks lets you build resilient asynchronous systems when you design for specific queue execution behaviors and service limits. You can prevent processing bottlenecks and unexpected delays by accounting for at-least-once delivery, system throttling, and target resource capacity.
Execution order
With the exception of tasks scheduled to run in the future, task queues are completely platform-independent about execution order. There are no guarantees or best-effort attempts made to execute tasks in any particular order. Specifically: there are no guarantees that old tasks will execute unless a queue is completely emptied. A number of common cases exist where newer tasks are executed sooner than older tasks, and the patterns surrounding this can change without notice.
Execution delay
Cloud Tasks executions might occasionally experience minor delays, usually lasting a few minutes, due to internal system restarts. Tasks are delayed, but no tasks are lost. These are system-level events with no workaround. These events aren't logged, and their occurrence has no set timeline.
Duplicate execution
Cloud Tasks aims for a strict "execute exactly once" semantic. However, in situations where a design trade-off must be made between guaranteed execution and duplicate execution, the service errs on the side of guaranteed execution. As such, a non-zero number of duplicate executions do occur. Ensure that your app handles duplicate executions gracefully so that they don't cause unexpected failures. In production, more than 99.999% of tasks are executed only once.
Resource limitations
The most common source of backlogs in immediate processing queues is exhausting resources on the target instances. If you attempt to execute 100 tasks per second on frontend instances that can only process 10 requests per second, a backlog builds. This typically manifests in one of two ways, either of which can generally be resolved by increasing the number of instances processing requests.
Backoff errors and enforced rates
Servers that are being overloaded can start to return backoff errors: HTTP 503
(for App Engine targets), or HTTP 429 or 5xx (for external targets).
Cloud Tasks reacts to these errors by slowing down execution until
errors stop. This system throttling prevents the worker from overloading. Your
settings aren't changed.
System throttling happens in the following circumstances:
Cloud Tasks backs off on all errors. Normally, Cloud Tasks uses the backoff specified in
retry_config. However, if the worker returns HTTP429 Too Many Requests,503 Service Unavailable, or5xxerror responses, or if the error rate is high, Cloud Tasks enforces a lower dispatch rate and considers the retry delay specified in theRetry-AfterHTTP response header. After errors stop and target handlers return HTTP2xxresponses, the queue gradually ramps its enforced dispatch rate back up.To prevent traffic spikes and smooth sudden increases in load, Cloud Tasks gradually ramps up dispatches when a queue is newly created, resumed after being paused, or idle after even a few minutes without active dispatches. This ramp-up also occurs when a large number of tasks suddenly becomes available to dispatch, such as during task creation spikes or when many tasks are scheduled for the same time. From a cold or idle state, Cloud Tasks starts dispatching at a low baseline rate and progressively steps up the enforced dispatch rate across periodic evaluation intervals. The rate continues to step up toward your configured
max_dispatches_per_secondrate as long as queue demand remains saturated and target handlers return HTTP2xxresponses. For more information about cold or idle ramp-up and managing resumed queues, see Initial queue ramp-up and Target overload.
Latency spikes and max concurrent
Overloaded servers can also respond with large increases in latency.
In this situation, requests remain open for longer. Because queues run with a
maximum concurrent number of tasks, this can result in queues being unable to
execute tasks at the expected rate. Increasing the
max_concurrent_dispatches
value for the affected queues can help in situations where the value has been
set too low, introducing an artificial rate limit. But increasing
max_concurrent_dispatches is unlikely to relieve any underlying resource
pressure.
Ramp-up issues with long-running tasks
Cloud Tasks queues ramp up their output in part based on the number of successful previously dispatched tasks. If the task handler takes a considerable period of time—on the order of minutes—to complete a task and return a success response, there can be a lag in the queue ramp-up rate.
Viewing more than 5,000 tasks
If you have more than 5,000 tasks, some tasks aren't visible in the Google Cloud console. Use gcloud CLI to view all tasks.
Maximum reported queue depth metric
The maximum queue depth metric reported by Cloud Tasks is capped at 1,000,000 tasks. This is intended to improve the performance of underlying task workers and there is no impact to the number of tasks that can be sent to or processed by your queue. Tasks sent to the queue above the queue depth limit continue to execute as intended.
To fetch the current queue depth beyond 1,000,000 tasks, you can use the
queues.tasks.list method.
This method returns all tasks with pagination, letting you aggregate the data,
and perform a count operation. However, depending on the queue depth size, the
method might encounter quota restrictions.
Recreating a queue with the same name
Deleting a queue permanently and irreversibly deletes all tasks in the queue, including scheduled tasks and tasks that are waiting to execute or retry. Neither you nor Cloud Customer Care can recover deleted queues or tasks.
After you delete a queue, you must wait 3 days before recreating it with the same name. This waiting period prevents unexpected behavior in tasks that are executing at the time of deletion and avoids internal process failures during the delete and recreate cycle. The waiting period applies to queues deleted through any interface, including the Google Cloud console, the gcloud CLI, the REST or RPC API, or Terraform.
During this tombstone window of up to 3 days, a queue creation request might
appear to succeed without recreating the queue, and uploading a
queue.yaml file that includes the deleted
queue name might fail. To confirm that a queue with the same name was recreated,
retrieve the queue using the
gcloud tasks queues describe
command or the
projects.locations.queues.get
method. A successful (200 OK) response confirms that the queue was recreated.
To clear tasks or modify a queue without losing the queue or triggering the
3-day waiting period, purge all tasks from the
queue, update the queue in
place, or pause the
queue instead of deleting and
recreating it. If you manage queues with Terraform, set prevent_destroy = true
in the lifecycle block to prevent accidental queue deletion.
Unsupported target when using a secure perimeter
If you have set up a secure perimeter using VPC Service Controls, HTTP requests from a
Cloud Tasks execution are blocked for unsupported targets, and
fail with a TARGET_TYPE_NOT_PERMITTED_FOR_VPC error code. For more
information, see Set up a service perimeter using
VPC Service Controls.
Resource locations constraints
Cloud Tasks supports restricting resource locations; however, limitations apply to the following regions:
us-central1us-central2(private Google Cloud region)
If you specify either region in your organization policy, you must include both
us-central1 and us-central2, even if you aren't creating
Cloud Tasks resources in both regions. You can include the region
us-central2 in your organization policy even if your organization doesn't use
private regions.