Building Foundations for AI products using dbt & Elementary

The Starting Point: Challenges in Data Management

In April 2024, I assumed leadership of the Data function at Dooly. We had a strong team of highly motivated individuals, but my initial assessments revealed a few technical limitations. Our existing data warehouse implementation, characterized by its narrow scope and reliance on DBT only for transformation tasks, could have been more optimal. Our biggest challenge as a small data team was balancing the business's current needs with advancing our data maturity for the future.

Ensuring data quality and discerning an error versus an actual observed anomaly were holding us back. The team was constantly bogged down with code change requests and data quality inquiries, and we only knew if a metric behaved oddly if a stakeholder told us. This is because we don’t look at all our metrics daily; we only look at a subset.

Compounding these issues was a notable absence of comprehensive documentation and minimal use of unit testing. Consequently, making even fundamental modifications to the codebase was painstakingly slow.

Identifying the Need to Rebuild

It became increasingly evident that a radical transformation was necessary to unlock the full potential of our data and foster innovation. We aimed to enhance our ability to manage data quality and reliability significantly. This realization kickstarted an ambitious data warehousing project to overhaul our data infrastructure from the ground up.

Implementing Data Observability with Elementary

In our quest for a more efficient and reliable data infrastructure, Elementary emerged as a crucial tool. Recognizing the imperative for an observability solution, Elementary was integrated into our ecosystem.

Implementing Elementary was relatively easy for us. It took about three days to onboard the tool, decide on appropriate testing, and set up Slack and email testing. Elementary allowed us to centralize our testing and anomaly testing and alert us of any data oddities. Sometimes, a significant spike or drop in a metric is genuine and warrants investigation; other times, it is due to incorrect data.

Adding Elementary proved to be exceptionally beneficial: 

1. Testing coverage for our DBT models has increased to 90% of all models.
2. Support messages due to the business detecting incorrect data have dropped 75%
3. The time it takes to identify anomalies is immeasurable. We would only know a metric has become volatile once someone observes it. Elementary has reduced this.
4. Our data team now spends less time reviewing metrics, and we focus on new data products.

Most importantly, this strategic addition aimed to shift our team's focus toward developing new data products rather than being bogged down by monitoring what occurred in the past.

Building a Foundation for AI

The most significant outcome of this transformation has been establishing a robust foundation for innovation. This foundation has been instrumental in launching an AI function at Dooly, a collaborative effort between our Data and Engineering teams.

I believe in data maturity as a path. To have an effective self-service BI platform, you must have well-defined data and clean data marts to track KPIs and metrics. Therefore, our data org must master the more straightforward activities to move into machine learning and more advanced techniques. 

Below is an excellent Tim Elliot blog graph describing data maturity.

‍
Elementary lets us ensure that our data is well-behaved and defined. We must measure the quality and anomalies of a KPI with it being represented correctly. Similarly, if a generative AI feature depends on user data to fine-tune, the data must be well-defined and high-quality.

Our AI initiatives' successes directly resulted from the groundwork laid by our data warehousing project, Elementary being a part of it.

Plans for the future: Increasing collaboration and trust

With solid business results, Dooly plans to expand its data organization. With additional resources, we want Elementary to be a cornerstone of our data observability for our analytics engineering efforts.

We will have a hybrid data org where analytics and engineers collaborate closely with various business units. This will allow us to build strong relationships with the business. We plan to use Elementary to send reports to each business unit. This way, the business and data orgs have mutual trust and ownership in the quality of their insights.