Document AI generates evaluation metrics, such as precision and recall, to help
you determine the predictive performance of your processors.

These evaluation metrics are generated by comparing the entities returned by the
processor (the predictions) against the annotations in the test documents. If
your processor does not have a test set, then you must first
[create a dataset](https://docs.cloud.google.com/document-ai/docs/create-dataset) and [label the test documents](https://docs.cloud.google.com/document-ai/docs/label-documents).

## Run an evaluation

An evaluation is automatically run whenever you train or uptrain a processor
version.

You can also manually run an evaluation. This is required to generate updated
metrics after you've modified the test set, or if you are evaluating a
pretrained processor version.

> [!NOTE]
> **Note:** Document AI cannot and does not calculate evaluation metrics for a label if the processor version cannot extract that label (for example, the label was disabled at the time of training) or if the test set does not include annotations for that label. Such labels are not included in aggregated metrics.

### Web UI

1. In the Google Cloud console, go to the **Processors** page and choose your
   processor.

   [Go to the Processors page](https://console.cloud.google.com/ai/document-ai/processors)
2. In the **Evaluate \& Test** tab, select the **Version** of the processor to
   evaluate and then click **Run new evaluation**.

Once complete, the page contains evaluation metrics for all labels and for each
individual label.

### Python


For more information, see the
[Document AI Python API
reference documentation](https://docs.cloud.google.com/python/docs/reference/documentai/latest).


To authenticate to Document AI, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment).


    from google.api_core.client_options import ClientOptions
    from google.cloud import documentai  # type: ignore

    # TODO(developer): Uncomment these variables before running the sample.
    # project_id = 'YOUR_PROJECT_ID'
    # location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'
    # processor_id = 'YOUR_PROCESSOR_ID'
    # processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'
    # gcs_input_uri = # Format: gs://bucket/directory/


    def evaluate_processor_version_sample(
        project_id: str,
        location: str,
        processor_id: str,
        processor_version_id: str,
        gcs_input_uri: str,
    ) -> None:
        # You must set the api_endpoint if you use a location other than 'us', e.g.:
        opts = ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")

        client = documentai.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html(client_options=opts)

        # The full resource name of the processor version
        # e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`
        name = client.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html#google_cloud_documentai_v1_services_document_processor_service_DocumentProcessorServiceClient_processor_version_path(
            project_id, location, processor_id, processor_version_id
        )

        evaluation_documents = documentai.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.types.BatchDocumentsInputConfig.html(
            gcs_prefix=documentai.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.types.GcsPrefix.html(gcs_uri_prefix=gcs_input_uri)
        )

        # NOTE: Alternatively, specify a list of GCS Documents
        #
        # gcs_input_uri = "gs://bucket/directory/file.pdf"
        # input_mime_type = "application/pdf"
        #
        # gcs_document = documentai.GcsDocument(
        #     gcs_uri=gcs_input_uri, mime_type=input_mime_type
        # )
        # gcs_documents = [gcs_document]
        # evaluation_documents = documentai.BatchDocumentsInputConfig(
        #     gcs_documents=documentai.GcsDocuments(documents=gcs_documents)
        # )
        #

        request = documentai.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.types.EvaluateProcessorVersionRequest.html(
            processor_version=name,
            evaluation_documents=evaluation_documents,
        )

        # Make EvaluateProcessorVersion request
        # Continually polls the operation until it is complete.
        # This could take some time for larger files
        operation = client.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html#google_cloud_documentai_v1_services_document_processor_service_DocumentProcessorServiceClient_evaluate_processor_version(request=request)
        # Print operation details
        # Format: projects/PROJECT_NUMBER/locations/LOCATION/operations/OPERATION_ID
        print(f"Waiting for operation {operation.operation.name} to complete...")
        # Wait for operation to complete
        response = documentai.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.types.EvaluateProcessorVersionResponse.html(operation.result())

        # After the operation is complete,
        # Print evaluation ID from operation response
        print(f"Evaluation Complete: {response.evaluation}")

### Get results of an evaluation

### Web UI

1. In the Google Cloud console, go to the **Processors** page and choose your
   processor.

   [Go to the Processors page](https://console.cloud.google.com/ai/document-ai/processors)
2. In the **Evaluate \& Test** tab, select the **Version** of the processor to view evaluation.

Once complete, the page contains evaluation metrics for all labels and for each
individual label.

### Python


For more information, see the
[Document AI Python API
reference documentation](https://docs.cloud.google.com/python/docs/reference/documentai/latest).


To authenticate to Document AI, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment).


    from google.api_core.client_options import ClientOptions
    from google.cloud import documentai  # type: ignore

    # TODO(developer): Uncomment these variables before running the sample.
    # project_id = 'YOUR_PROJECT_ID'
    # location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'
    # processor_id = 'YOUR_PROCESSOR_ID' # Create processor before running sample
    # processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'
    # evaluation_id = 'YOUR_EVALUATION_ID'


    def get_evaluation_sample(
        project_id: str,
        location: str,
        processor_id: str,
        processor_version_id: str,
        evaluation_id: str,
    ) -> None:
        # You must set the api_endpoint if you use a location other than 'us', e.g.:
        opts = ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")

        client = documentai.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html(client_options=opts)

        # The full resource name of the evaluation
        # e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`
        evaluation_name = client.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html#google_cloud_documentai_v1_services_document_processor_service_DocumentProcessorServiceClient_evaluation_path(
            project_id, location, processor_id, processor_version_id, evaluation_id
        )
        # Make GetEvaluation request
        evaluation = client.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html#google_cloud_documentai_v1_services_document_processor_service_DocumentProcessorServiceClient_get_evaluation(name=evaluation_name)

        create_time = evaluation.create_time
        document_counters = evaluation.document_counters

        # Print the Evaluation Information
        # Refer to https://cloud.google.com/document-ai/docs/reference/rest/v1beta3/projects.locations.processors.processorVersions.evaluations
        # for more information on the available evaluation data
        print(f"Create Time: {create_time}")
        print(f"Input Documents: {document_counters.input_documents_count}")
        print(f"\tInvalid Documents: {document_counters.invalid_documents_count}")
        print(f"\tFailed Documents: {document_counters.failed_documents_count}")
        print(f"\tEvaluated Documents: {document_counters.evaluated_documents_count}")

### List all evaluations for a processor version

### Python


For more information, see the
[Document AI Python API
reference documentation](https://docs.cloud.google.com/python/docs/reference/documentai/latest).


To authenticate to Document AI, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment).


    from google.api_core.client_options import ClientOptions
    from google.cloud import documentai  # type: ignore

    # TODO(developer): Uncomment these variables before running the sample.
    # project_id = 'YOUR_PROJECT_ID'
    # location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'
    # processor_id = 'YOUR_PROCESSOR_ID' # Create processor before running sample
    # processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'


    def list_evaluations_sample(
        project_id: str, location: str, processor_id: str, processor_version_id: str
    ) -> None:
        # You must set the api_endpoint if you use a location other than 'us', e.g.:
        opts = ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")

        client = documentai.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html(client_options=opts)

        # The full resource name of the processor version
        # e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`
        parent = client.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html#google_cloud_documentai_v1_services_document_processor_service_DocumentProcessorServiceClient_processor_version_path(
            project_id, location, processor_id, processor_version_id
        )

        evaluations = client.https://docs.cloud.google.com/python/docs/reference/documentai/latest/google.cloud.documentai_v1.services.document_processor_service.DocumentProcessorServiceClient.html#google_cloud_documentai_v1_services_document_processor_service_DocumentProcessorServiceClient_list_evaluations(parent=parent)

        # Print the Evaluation Information
        # Refer to https://cloud.google.com/document-ai/docs/reference/rest/v1beta3/projects.locations.processors.processorVersions.evaluations
        # for more information on the available evaluation data
        print(f"Evaluations for Processor Version {parent}")

        for evaluation in evaluations:
            print(f"Name: {evaluation.name}")
            print(f"\tCreate Time: {evaluation.create_time}\n")

## Evaluation metrics for all labels

![evaluate-the-performance-of-processors-1](https://docs.cloud.google.com/document-ai/docs/images/manage/evaluate-the-performance-of-processors-1.png)

Metrics for **All labels** are computed based on the number of true positives,
false positives, and false negatives in the dataset across all labels, and thus,
are weighted by the number of times each label appears in the dataset. For
definitions of these terms, see
[Evaluation metrics for individual labels](https://docs.cloud.google.com/document-ai/docs/evaluate#individual-labels).

- **Precision:** the proportion of predictions that match the annotations in
  the test set. Defined as `True Positives / (True Positives + False
  Positives)`

- **Recall:** the proportion of annotations in the test set that are correctly
  predicted. Defined as `True Positives / (True Positives + False Negatives)`

- **F1 score:** the harmonic mean of precision and recall, which combines
  precision and recall into a single metric, providing equal weight to both.
  Defined as `2 * (Precision * Recall) / (Precision + Recall)`

> [!NOTE]
> **Note:** Document AI does not provide a metric for Accuracy. The accuracy metric, often defined as the proportion of instances that are predicted correctly, is less meaningful because 1) not all labels appear in the test set (for example, optional fields) and 2) there may be multiple values for a single label (for example, line items in an invoice). F1, on the other hand, can be considered roughly equivalent to accuracy, but it more meaningfully accommodates these scenarios.

## Evaluation metrics for individual labels

![evaluate-the-performance-of-processors-2](https://docs.cloud.google.com/document-ai/docs/images/manage/evaluate-the-performance-of-processors-2.png)

- **True Positives:** the predicted entities that match an annotation in the
  test document. For more information, see
  [matching behavior](https://docs.cloud.google.com/document-ai/docs/evaluate#matching_behavior).

- **False Positives:** the predicted entities that don't match any annotation
  in the test document.

- **False Negatives:** the annotations in the test document that don't match
  any of the predicted entities.

  - **False Negatives (Below Threshold):** the annotations in the test document that would have matched a predicted entity, but the predicted entity's confidence value is below the specified confidence threshold.

> [!NOTE]
> **Note:** False positives and false negatives are not mutually exclusive. For example, if a predicted entity has a corresponding, albeit incorrect, annotation in the test set, then there is one false positive for the predicted entity and one false negative for the annotation. A predicted entity without an annotation is only a false positive. An annotation without an associated prediction is only a false negative.

> [!TIP]
> **Tip:** If the quality of `checkbox` extraction is not high enough, consider running the documents through the [Form Parser](https://docs.cloud.google.com/document-ai/docs/processors-list#processor_form-parser). The Form Parser extracts information that could potentially boost the quality of checkbox extraction. Specifically, Form Parser can automatically draw bounding boxes around checkboxes whereas manually drawn bounding boxes might be less consistent. To do this, process your documents using the Form Parser, then [import](https://docs.cloud.google.com/document-ai/docs/create-dataset#import) the processed documents and annotate them again.

### Confidence threshold

The evaluation logic ignores any predictions with confidence below the specified
**Confidence Threshold** , even if the prediction is correct. Document AI
provides a list of *False Negatives (Below Threshold)*, which are the
annotations that would have a match if the confidence threshold were set lower.

Document AI automatically computes the **optimal threshold**, which maximizes
the F1 score, and by default, sets the confidence threshold to this optimal
value.

> [!NOTE]
> **Note:** If there are multiple optimal threshold values (with the same maximal F1 score), the lowest value is chosen.

You are free to choose your own confidence threshold by moving the slider bar.
In general, higher confidence threshold results in:

- higher precision, because the predictions are more likely to be correct.
- lower recall, because there are fewer predictions.

### Tabular entities

The metrics for a parent label are not calculated by directly averaging the
child metrics, but rather, by applying the parent's confidence threshold to all
of its child labels and aggregating the results.

The optimal threshold for the parent is the
[confidence threshold](https://docs.cloud.google.com/document-ai/docs/evaluate#confidence_threshold) value that, when applied to all
children, yields the maximum F1 score for the parent.

## Matching behavior

A predicted entity matches an annotation if:

- the type of the predicted entity ([`entity.type`](https://docs.cloud.google.com/document-ai/docs/reference/rest/v1/Document#entity)) matches the annotation's label name
- the value of the predicted entity ([`entity.mention_text`](https://docs.cloud.google.com/document-ai/docs/reference/rest/v1/Document#Entity) or [`entity.normalized_value.text`](https://docs.cloud.google.com/document-ai/docs/reference/rest/v1/Document#Entity)) matches the annotation's text value, subject to [fuzzy matching](https://docs.cloud.google.com/document-ai/docs/evaluate#fuzzy_matching) if it is enabled.

Note that type and text value are all that is used for matching. Other
information, such as text anchors and bounding boxes (with the exception of
tabular entities described below) are not used.

> [!NOTE]
> **Note:** Some [Identity processors](https://docs.cloud.google.com/document-ai/docs/processors-list#identity_processors) extract a *Portrait* entity, which consists of no text and only a bounding box that indicates where a person's portrait is in an identity document. Because it lacks text, portrait labels and entities are not evaluated.

### Single- versus multi-occurrence labels

Single-occurrence labels have one value per document (for example, invoice ID)
even if that value is annotated multiple times in the same document (for
example, the invoice ID appears in every page of the same document). Even if the
multiple annotations have different text, they are considered equal. In other
words, if a predicted entity matches any of the annotations, it counts as a
match. The extra annotations are considered *duplicate mentions* and don't
contribute towards any of the true positive, false positive, or false negative
counts.

Multi-occurrence labels can have multiple, different values. Thus, each
predicted entity and annotation is considered and matched separately. If a
document contains N annotations for a multi-occurrence label, then there can be
N matches with the predicted entities. Each predicted entity and annotation are
independently counted as a true positive, false positive, or false negative.

### Fuzzy Matching

The **Fuzzy Matching** toggle lets you tighten or relax some of the matching
rules to decrease or increase the number of matches.

For example, without fuzzy matching, the string `ABC` does not match `abc` due
to capitalization. But with fuzzy matching, they match.

When fuzzy matching is enabled, here are the rule changes:

- **Whitespace normalization:** removes leading-trailing whitespace and
  condenses consecutive intermediate whitespaces (including newlines) into
  single spaces.

- **Leading/trailing punctuation removal:** removes the following
  leading-trailing punctuation characters `!,.:;-"?|`.

- **Case-insensitive matching:** converts all characters to lowercase.

- **Money normalization:** For labels with the data type `money`, remove the
  leading-trailing currency symbols.

> [!NOTE]
> **Note:** You cannot use fuzzy matching on numeric values. For example, `1` and `1.00` don't match, even when fuzzy matching is enabled.

### Tabular entities

Parent entities and annotations don't have text values and are matched based on
the combined bounding boxes of their children. If there is only one predicted
parent and one annotated parent, they are automatically matched, regardless of
bounding boxes.

Once parents are matched, their children are matched as if they were non-tabular
entities. If parents are not matched, Document AI won't attempt to match
their children. This means that child entities can be considered
incorrect, even with the same text contents, if their parent entities are not
matched.

Parent / child entities are a Preview feature and only supported for tables with one layer of nesting.

## Export evaluation metrics

1. In the Google Cloud console, go to the **Processors** page and choose your
   processor.

   [Go to the Processors page](https://console.cloud.google.com/ai/document-ai/processors)
2. In the **Evaluate \& Test** tab, click **Download Metrics**, to download the
   evaluation metrics as a JSON file.

> [!NOTE]
> **Note:** The exported metrics are based on the value of the [Confidence threshold](https://docs.cloud.google.com/document-ai/docs/evaluate#confidence_threshold) slider at the time of export.

## Monitoring dashboard

The monitoring dashboard in the Google Cloud console provides a useful way to create
your own monitoring visualizations for different metrics and resources used in
Document AI processors.

List of metrics and their fields: **Number of successfully processed pages** and
**Total number of docs sent to processing** both have the following fields:
`location`, `processor type`, `processor_id`, `processor_version_id`, `status`,
and `processing_type`.

**Sync processing latency** , **Batch processing latency per document** , and
**Batch processing operation count** have the following fields: `location`,
`processor_type`, `processor_id`, `processor_version_id`, and `status`.

### Create a monitoring view

You can monitor your processors individually and collectively, with a per
processor view and a per project view. The steps for these are the same.

1. In the Google Cloud console, in the Document AI section, go to the
   **Processors** page.

   [Go to the Processors page](https://console.cloud.google.com/ai/document-ai/processors)
2. Optional: If you want to monitor a specific processor, select it from the list.

3. Use the navigation pane to select the **Monitoring** monitoring option.

4. Set the **Time range**.

5. Fill the checkboxes for the selected metric and resource labels, and select
   **Ok**.

   > [!TIP]
   > **Tip:** You can hold the pointer over the trendlines and series legends to highlight them.

6. Optional: You can see more options for each graph with the **Menu** option.