This document describes how to use [Knowledge Catalog](https://docs.cloud.google.com/knowledge-catalog/docs/introduction.md.txt) to view metadata of Dataform repositories, annotate repositories with aspects, and track data lineage for Dataform workflow executions.

<br />

Knowledge Catalog lets you store and manage your metadata.
Dataform repositories are available in Knowledge Catalog
by default, without any additional configuration.
Knowledge Catalog automatically retrieves
the following metadata from Dataform repositories:

- Data asset name

- Data asset parent

- Data asset location

- Data asset type

- Corresponding Google Cloud project

- Third-party repository source

- Service account

## Before you begin

1. In the Google Cloud console, on the project selector page,
   select or create a Google Cloud project.

   **Roles required to select or create a project**
   - **Select a project**: Selecting a project doesn't require a specific IAM role---you can select any project that you've been granted a role on.
   - **Create a project** : To create a project, you need the Project Creator role (`roles/resourcemanager.projectCreator`), which contains the `resourcemanager.projects.create` permission. [Learn how to grant
     roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access).

   > [!NOTE]
   > **Note**: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   [Go to project selector](https://console.cloud.google.com/projectselector2/home/dashboard)
2.


   Enable the Dataplex API and Data Lineage API APIs, if any are not already enabled.


   **Roles required to enable APIs**


   To enable APIs, you need the `serviceusage.services.enable` permission. If you
   created the project, then you likely already have this permission through the
   Owner role (`roles/owner`). Otherwise, you can get this permission through the
   Service Usage Admin role (`roles/serviceusage.serviceUsageAdmin`).
   [Learn how to grant roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

   [Enable the APIs](https://console.cloud.google.com/apis/enableflow?apiid=dataplex.googleapis.com,datalineage.googleapis.com)
3. Ensure that you have the
   [necessary Identity and Access Management (IAM) permissions](https://docs.cloud.google.com/dataform/docs/use-dataplex.md.txt#required-roles).

### Required roles

To manage Dataform repositories and view data lineage in
Knowledge Catalog, you need the
[predefined Dataform roles](https://docs.cloud.google.com/dataform/docs/access-control.md.txt#predefined-roles)
that give you access to Dataform resources, and the required
[Knowledge Catalog roles](https://docs.cloud.google.com/knowledge-catalog/docs/iam-roles.md.txt).

The following table lists the roles that each principal needs to record and
view Dataform data lineage:

| Task | Principal | Required role | Where to grant the role |
|---|---|---|---|
| Record lineage from workflow invocations | The [custom service account](https://docs.cloud.google.com/dataform/docs/access-control.md.txt#about-service-accounts) of the repository, release configuration, or workflow configuration. If no custom service account is set, the [Dataform service agent](https://docs.cloud.google.com/dataform/docs/access-control.md.txt#about-service-accounts) (`service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com`). | [Data Lineage Events Producer](https://docs.cloud.google.com/iam/docs/roles-permissions/datalineage.md.txt#datalineage.producer) (`roles/datalineage.producer`) | The project that contains the repository |
| Record lineage from the Dataform CLI | The [Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials.md.txt) that the CLI uses | Data Lineage Events Producer (`roles/datalineage.producer`) | The project of each output table |
| View the lineage graph | The user who views lineage | [Data Lineage Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/datalineage.md.txt#datalineage.viewer) (`roles/datalineage.viewer`) | The project where lineage is recorded and the project where you view lineage |
| Open Dataform from the lineage graph | The user who views lineage | [Dataform Viewer](https://docs.cloud.google.com/dataform/docs/access-control.md.txt#dataform.viewer) (`roles/dataform.viewer`), or the `dataform.repositories.get` and `dataform.workflowInvocations.get` permissions | The project that contains the repository |
| View BigQuery table details | The user who views lineage | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) (`roles/bigquery.dataViewer`), or the `bigquery.tables.get` permission | The project that stores the table |
| View BigQuery job details | The user who views lineage | [BigQuery Resource Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.resourceViewer) (`roles/bigquery.resourceViewer`), or the `bigquery.jobs.get` permission | The project that runs the job |

The Data Lineage Events Producer role contains the
`datalineage.locations.processOpenLineageMessage` permission, which is required
to record lineage.

> [!NOTE]
> **Note:** In the lineage graph, the **Open in Dataform** link appears even if you don't have the Dataform Viewer role (`roles/dataform.viewer`). If you click **Open in Dataform** without the required Dataform permissions, the Google Cloud console returns a permission error.

## Manage metadata in Knowledge Catalog

Knowledge Catalog logs Dataform repositories as
[entries](https://docs.cloud.google.com/dataplex/docs/ingest-custom-sources#entries) with the following
entry values:

System entry group
:   The [system entry group](https://docs.cloud.google.com/dataplex/docs/ingest-custom-sources#entry-groups)
    for Dataform repositories is `@dataform`. To view details of
    repository entries in Knowledge Catalog, you need to view the `dataform`
    system entry group.
    For instructions about how to view a list of all entries in an entry group, see
    [View details of an entry group](https://docs.cloud.google.com/dataplex/docs/ingest-custom-sources#entry-group-details)
    in the Knowledge Catalog documentation.

System entry type
:   The [system entry type](https://docs.cloud.google.com/dataplex/docs/ingest-custom-sources#entry-types)
    for Dataform repositories is `dataform-repository`.
    To view details of Dataform repositories,
    you need to view the `dataform-repository` system entry type.
    For instructions about how to view details of a selected entry type, see
    [View details of an entry type](https://docs.cloud.google.com/dataplex/docs/ingest-custom-sources#entry-type-details)
    in the Knowledge Catalog documentation.
    For instructions about how to view details of a selected entry, see
    [View details of an entry](https://docs.cloud.google.com/dataplex/docs/search-assets#view-entry-details)
    in the Knowledge Catalog documentation.

### Search for assets

For instructions about how to search for data assets in Knowledge Catalog, see
[Search for data assets in Knowledge Catalog](https://docs.cloud.google.com/dataplex/docs/search-assets)
in the Knowledge Catalog documentation.

## Annotate repositories

To provide additional context to Dataform repositories in
Knowledge Catalog, you can annotate Dataform entries
with [aspects](https://docs.cloud.google.com/dataplex/docs/enrich-entries-metadata#aspects).

The system aspect type for Dataform repositories
is `dataform-repository`.

For instructions about how to annotate entries with aspects, see
[Manage aspects and enrich metadata](https://docs.cloud.google.com/dataplex/docs/enrich-entries-metadata)
in the Knowledge Catalog documentation.

## Track data lineage in Knowledge Catalog

Dataform reports [data lineage](https://docs.cloud.google.com/knowledge-catalog/docs/about-data-lineage.md.txt)
to Knowledge Catalog for the actions that it runs in
Dataform workflows and BigQuery pipelines. For
each executed action, the lineage graph in Knowledge Catalog and
BigQuery shows a Dataform process between the
tables that the action reads and the table that it writes. You don't need to
change your SQLX files to report lineage.

You can use Dataform data lineage for the following tasks:

- **Impact analysis**: identify every downstream action and table before you
  change a table.

- **Root-cause analysis**: trace incorrect data back to the action and the
  workflow invocation that wrote it.

- **Governance**: show how data flows across projects and which pipeline moved
  the data.

- **End-to-end lineage**: view Dataform lineage in one graph
  together with lineage from BigQuery, Managed Service for Apache Airflow,
  Managed Service for Apache Spark, and Dataflow.

### How data lineage works

Dataform sends an [OpenLineage](https://openlineage.io/) `START`
event to the Data Lineage API when an action begins. When the action ends,
Dataform sends a `COMPLETE`, `FAIL`, or `ABORT` event. Each
Dataform action appears in Knowledge Catalog as a process
with the `BIGQUERY_PIPELINES` source type, and each execution of the action
appears as a run of that process.

The following table describes which Dataform action types report
lineage:

| Action type | Reports lineage |
|---|---|
| Table, view, incremental table, materialized view, assertion, property graph, data preparation, and operation with `hasOutput: true` | Yes. Reports the process, run, and table-to-table lineage links. |
| Operation without `hasOutput: true` | Process and run only. Inputs and outputs are empty, so no table-to-table lineage link is created. |
| Declaration | No. Declarations appear only as inputs to other actions. |
| Notebook, SQL unit test | No |

Reporting lineage never causes a workflow invocation to fail. If the
Data Lineage API is disabled, a permission is missing, or a quota is
exceeded, Dataform skips lineage reporting and the workflow
invocation continues without changes.

Dataform writes lineage to the regional endpoint of the
BigQuery processing location of each action. If the regional
endpoint is unavailable, Dataform writes lineage to the global
endpoint.

### How Dataform workflows map to data lineage

The following table describes how Dataform execution entities
map to the
[Knowledge Catalog data lineage information model](https://docs.cloud.google.com/knowledge-catalog/docs/about-data-lineage.md.txt#information-model):

| Data lineage object | Dataform equivalent |
|---|---|
| Process | A Dataform action |
| Run | One execution of the action in a workflow invocation |
| Lineage event | A link from the input tables to the output table for that run |

### Set up data lineage

To set up Dataform to report data lineage, follow these steps:

1. Enable the Data Lineage API (`datalineage.googleapis.com`) in the project
   that contains the Dataform repository and in the project
   where users view lineage.

2. In the project that contains the repository, grant the
   [Data Lineage Events Producer role](https://docs.cloud.google.com/iam/docs/roles-permissions/datalineage.md.txt#datalineage.producer)
   (`roles/datalineage.producer`) to the identity that runs your workflows:

   - If the repository, release configuration, or workflow configuration uses
     a [custom service account](https://docs.cloud.google.com/dataform/docs/access-control.md.txt#about-service-accounts),
     grant the role to the custom service account.

   - If no custom service account is set, grant the role to the
     Dataform service agent
     (`service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com`).

Workflow invocations that run with user credentials for a Google Account don't
report lineage.

### Enable or disable data lineage

During the initial rollout period, data lineage reporting is opt-in.

To control data lineage reporting for a repository or BigQuery
pipeline, set the `lineage.enabled` property in the `workflow_settings.yaml`
file:

1. In the Google Cloud console, go to the **Dataform** page.

   [Go to Dataform](https://console.cloud.google.com/bigquery/dataform)
2. Select a repository, and then select a development workspace.

3. In the **Files** pane, open the `workflow_settings.yaml` file.

4. To turn on data lineage reporting, add the following `lineage` block:

   ```yaml
   lineage:
     enabled: true
   ```

   To opt the repository out of data lineage reporting after lineage is turned
   on by default, set `enabled` to `false`:

   ```yaml
   lineage:
     enabled: false
   ```
5. [Commit and push your changes](https://docs.cloud.google.com/dataform/docs/manage-repository.md.txt#commit-changes)
   to your repository.

To stop data lineage reporting for a whole project, disable the
Data Lineage API in the project. Disabling the API also stops data lineage
reporting from other services in the project.

### View data lineage

To view data lineage for a table that a Dataform action
produces, follow these steps:

1. In the Google Cloud console, go to the **BigQuery** page.

   [Go to BigQuery](https://console.cloud.google.com/bigquery)
2. In the **Explorer** pane, expand your project and dataset, and then select a
   table that a Dataform action produces.

3. Click the **Lineage** tab.

4. To view the action, the repository, and the runs, select the
   Dataform process node.

5. To open the repository in Dataform, click
   **Open in Dataform**.

For more information about navigating lineage graphs, see
[View data lineage in Knowledge Catalog](https://docs.cloud.google.com/knowledge-catalog/docs/use-lineage.md.txt).

### Report data lineage from the Dataform CLI

To report data lineage from local or CI runs of the
[Dataform CLI](https://docs.cloud.google.com/dataform/docs/use-dataform-cli.md.txt), set
`lineage.enabled: true` in `workflow_settings.yaml`, or run the following
command:

```bash
dataform run --emit-lineage
```

The `--emit-lineage` and `--no-emit-lineage` flags override the
`lineage.enabled` setting for a single run.

The Dataform CLI uses
[Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials.md.txt)
and records lineage in the project of each output table. In each of these
projects, enable the Data Lineage API and grant the Data Lineage Events
Producer role (`roles/datalineage.producer`) to the credential. The CLI writes
lineage messages to `stderr` with the `[lineage]` prefix.

For more information about reporting lineage from the CLI, see the
[Dataform CLI reference on GitHub](https://github.com/dataform-co/dataform/blob/main/cli/README.md#emitting-lineage-events-for-a-run).

### Troubleshoot data lineage

The following table describes common data lineage issues and how to fix them:

| Symptom | Likely cause | Fix |
|---|---|---|
| The **Lineage** tab doesn't show Dataform nodes. | The Data Lineage API is disabled. | Enable the API in the project that contains the repository and in the project where you view lineage. |
| A repository doesn't report lineage. | The identity that runs the workflow doesn't have the Data Lineage Events Producer role. | Grant `roles/datalineage.producer` to the custom service account or to the Dataform service agent. |
| Workflow invocations that run with user credentials don't report lineage. | Lineage reporting doesn't support user credentials. | Run the workflow with a custom service account. |
| A repository doesn't report lineage although the role is granted. | Dataform can't act as the custom service account. | Grant the Dataform service agent the [Service Account Token Creator role](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountTokenCreator) (`roles/iam.serviceAccountTokenCreator`) on the custom service account. |
| One repository doesn't report lineage. | The repository is opted out. | Remove `lineage.enabled: false` from `workflow_settings.yaml`. |
| Lineage is missing for some runs. | The Data Lineage API quota is exceeded. | Spread out workflow runs or request a higher quota. |
| CLI runs don't report lineage for some tables. | The API isn't enabled or the role isn't granted in the project of the output table. | Enable the API and grant the role in that project. |
| Clicking **Open in Dataform** returns a permission error. | The user doesn't have access to Dataform. | Grant the user the Dataform Viewer role (`roles/dataform.viewer`). |

After the first permission or API error in a workflow invocation,
Dataform stops reporting lineage for the rest of that workflow
invocation. The next workflow invocation tries to report lineage again.

When you contact support about a data lineage issue, include the workflow
invocation ID, the project ID, and the region.

#### View lineage logs for a workflow invocation

In Cloud Logging, Dataform writes lineage log entries to the
`dataform.googleapis.com/lineage_emission` log. Each entry includes the
`workflowInvocationId`, `actionTarget`, and `eventType` fields.

To view lineage logs for a workflow invocation, follow these steps:

1. In the Google Cloud console, open the details page of the workflow
   invocation.

2. Click **View logs**.

   Logging opens with logs filtered to the workflow invocation
   by `jsonPayload.workflowInvocationId`.

The `outcome` field in each lineage log entry has one of the following values:

- `SUCCEEDED`: the Data Lineage API accepted the event. These entries have
  `DEBUG` severity, and `eventType` is `START`, `COMPLETE`, `FAIL`, or `ABORT`.

- `SKIPPED`: Dataform skipped lineage reporting. These entries
  have `INFO` severity and are logged once for each reason in a workflow
  invocation. The `skipReason` field is `API_DISABLED`, `PERMISSION_DENIED`,
  `EUC_UNSUPPORTED`, or `AUTHORITY_UNAVAILABLE`, and the `errorMessage` field
  contains remediation steps.

- `FAILED`: lineage reporting failed after retries. These entries have
  `WARNING` severity and include the `errorCode` and `errorMessage` fields.

To filter for skipped or failed lineage reporting in Logs Explorer, run the
following query:

    log_id("dataform.googleapis.com/lineage_emission")
    jsonPayload.outcome=("SKIPPED" OR "FAILED")

#### Data lineage messages from the Dataform CLI

The Dataform CLI writes the following lineage messages to
`stderr`:

- `skip_reason=workflow_opt_out`: `lineage.enabled` is set to `false`.

- `skip_reason=invocation_override`: lineage reporting is turned off for this
  run with `--no-emit-lineage` or `--emit-lineage=false`.

- `skip_reason=dry_run`: dry runs don't report lineage.

- `skip_reason=api_disabled`: the Data Lineage API isn't enabled. The CLI
  doesn't send more lineage requests in this run.

- `skip_reason=permission_denied`: the credential doesn't have the Data Lineage
  Events Producer role (`roles/datalineage.producer`). The CLI doesn't send more
  lineage requests in this run.

- `Failed to emit lineage for action ...`: lineage reporting failed for one
  action. Lineage reporting for other actions continues.

## Limitations

Managing Dataform repositories and tracking data lineage in
Knowledge Catalog is subject to the following limitations:

- Only repository-level and workflow-level entries are available in
  Knowledge Catalog. Individual file-level code assets in
  Dataform, such as SQLX definition files, aren't cataloged as
  separate entries.

- Dataform reports table-level lineage only. Column-level lineage
  isn't reported.

- Declarations, notebooks, and SQL unit tests don't report lineage. Declarations
  appear only as inputs to other actions. Operations without `hasOutput: true`
  report the process and run only, without table-to-table lineage links.

- Workflow invocations that run with user credentials for a Google Account
  don't report lineage. To report lineage, use a custom service account.

- Data lineage reporting is subject to the
  [Data Lineage API quotas](https://docs.cloud.google.com/knowledge-catalog/docs/quotas.md.txt) for writes per
  project, identity, and region. At the standard tier, the quota is about 1,000
  actions per minute.

## Locations

You can use Knowledge Catalog to manage Dataform repositories
from all [Dataform locations](https://docs.cloud.google.com/dataform/docs/locations.md.txt).

## Quotas

Managing Dataform repositories in Knowledge Catalog
is subject to [Knowledge Catalog quotas and limits](https://docs.cloud.google.com/dataplex/docs/quotas).

## Pricing

Managing Dataform repositories in Knowledge Catalog
is subject to [Knowledge Catalog pricing](https://cloud.google.com/dataplex/pricing).

Data lineage ingestion is billed under the Knowledge Catalog Premium
Processing SKU, and only events that write table-to-table lineage links are
billed. Data lineage storage is billed under the metadata storage SKU. To
control data lineage costs, [opt workflows out](https://docs.cloud.google.com/dataform/docs/use-dataplex.md.txt#enable-data-lineage) in
`workflow_settings.yaml` or disable the Data Lineage API.

## What's next

- To learn more about Knowledge Catalog, see [Knowledge Catalog overview](https://docs.cloud.google.com/dataplex/docs/introduction) in the Knowledge Catalog documentation.