How the dbt Package Works with the Cloud Platform

This video explains how Elementary's open-source dbt package provides anomaly detection through dbt tests and uploads metadata and test results to a dedicated schema in your data warehouse. It highlights how the Cloud platform accesses this metadata without needing access to raw data and includes integrations with major data warehouses to automate monitoring for data freshness and volume.

In Elementary, we have the open source dbt package that enables two main things. One is anomaly detection as dbt test, but it also serves as an SDK that uploads all the metadata via the on-run-end hooks at the end of the dbt invocations and it uploads automatically all the metadata logs test results into the Elementary schema.

So it saved in your warehouse in a separate Elementary schema. And what's cool about that is that it enables the Cloud to get read only access  to these tables, to this schema specifically, without asking for access to your raw schemas, raw data. And it enables the Cloud to be relying only on metadata that is being synced from the data warehouse to the Cloud Platform.

In addition to that, in the Cloud Platform, we also have native integrations with all the major data warehouses and databases. And specifically, we collect metadata from the information schemas and query or job histories that are available in the data warehouses in order to provide you automated monitors specifically for freshness and volume, which are not coupled into your pipelines.