Download artifacts from a Job Run¶

This is a guideline on how to download manifest.json and catalog.json from a Job Run by using dbt CLoud Administrative API, under the Retrieve Run Artifact endpoint. Therefore, we don't need to run dbt docs generate locally anymore.

In order to support dbt Cloud users, dbterd is now having multiple CLI options starting with --dbt-cloud to let us configure the connection to the complete dbt Cloud Job Run.

Prerequisites

You have a dbt Cloud account with Team and Enterprise plans 💰
You have a job or go create a new job with a single step 🏃
```
dbt docs generate
```
Make sure that you have at least 1 successful run ✅

1. Prepare the environment variables¶

Behind the scene, the API Endpoint will look like:

https://{host_url}/api/{api_version}/accounts/{account_id}/runs/{run_id}/artifacts/{path}

And the dbt Cloud's Job Rub will have the URL constructed as:

https://<host_url>/deploy/<account_id>/projects/irrelevant/runs/<run_id>

In the above:

URL Part	Environment Variable	CLI Option	Description
`host_url`	`DBTERD_DBT_CLOUD_HOST_URL`	`--dbt-cloud-host-url`	Host URL, also known as Access URL (Default to `cloud.getdbt.com`)
`account_id`	`DBTERD_DBT_CLOUD_ACCOUNT_ID`	`--dbt-cloud-account-id`	dbt Cloud Account ID
`run_id`	`DBTERD_DBT_CLOUD_RUN_ID`	`--dbt-cloud-run-id`	dbt Cloud successful job run ID
`api_version`	`DBTERD_DBT_CLOUD_API_VERSION`	`--dbt-cloud-api-version`	dbt Cloud API version (Default to `v2`)
`path`	`N/A`	`N/A`	Artifact relative file path. You don't need to care about this part as `dbterd` managed it already

Besides, we need another one which is very important, the service token:

Go to Account settings / Service tokens. Click + New token
Enter Service token name e.g. "ST_dbterd"
Click Add and select Job Admin permission. Optionally, select the right project or all by default
Click Save
Copy token & Pass it to the Environment Variable (DBTERD_DBT_CLOUD_SERVICE_TOKEN) or the CLI Option (--dbt-cloud-service-token)

Finally, fill in your_value and execute the (Linux or Macos) command below:

export DBTERD_DBT_CLOUD_SERVICE_TOKEN=your_value
export DBTERD_DBT_CLOUD_ACCOUNT_ID=your_value
export DBTERD_DBT_CLOUD_RUN_ID=your_value
export DBTERD_DBT_CLOUD_HOST_URL=your_value # optional, default = cloud.getdbt.com
export DBTERD_DBT_CLOUD_API_VERSION=your_value # optional, default = v2

Or in Powershell:

$env:DBTERD_DBT_CLOUD_SERVICE_TOKEN="your_value"
$env:DBTERD_DBT_CLOUD_ACCOUNT_ID="your_value"
$env:DBTERD_DBT_CLOUD_RUN_ID="your_value"
$env:DBTERD_DBT_CLOUD_HOST_URL="your_value" # optional, default = cloud.getdbt.com
$env:DBTERD_DBT_CLOUD_API_VERSION="your_value" # optional, default = v2

2. Generate ERD file¶

We're going to use --dbt-cloud option to tell dbterd to use dbt Cloud API with all above variables.

The command will be looks like:

dbterd run --dbt-cloud [-s <dbterd selection>]

Note

You can not use --dbt option together with --dbt-cloud

and then, here is the sample console log:

dbterd - INFO - Run with dbterd==1.0.0 (main.py:54)
dbterd - INFO - Using dbt project dir at: C:\Sources\dbterd (base.py:46)
dbterd - INFO - Downloading...[URL: https://hidden/api/v2/accounts/hidden/runs/hidden/artifacts/manifest.json] (dbt_cloud.py:68)
dbterd - INFO - Completed [status: 200] (dbt_cloud.py:71)
dbterd - INFO - Downloading...[URL: https://hidden/api/v2/accounts/hidden/runs/hidden/artifacts/catalog.json] (dbt_cloud.py:68)
dbterd - INFO - Completed [status: 200] (dbt_cloud.py:71)
dbterd - INFO - Using dbt artifact dir at: hidden (base.py:73)
dbterd - INFO - Collected 4 table(s) and 3 relationship(s) (test_relationship.py:59)
dbterd - INFO - C:\Sources\dbterd\target/output.dbml (base.py:170)

Voila! Happy ERD 🎉!