Event Freshness Tests vs. Automated Freshness Tests

This video explains the difference between event freshness anomalies and automated freshness. While a table may update regularly, the underlying event data could still be delayed, particularly for event tables with production queues. By providing a timestamp column, Elementary can detect when event times are behind schedule. Additionally, if strict SLAs are required, users can use existing dbt tests, like the dbt utils recency test, to ensure data is up to date. Elementary provides an IDE with prebuilt templates to simplify the process of setting up these tests.

Just to give you like some context on the difference between, for example, event freshness to the automated freshness. Some tables could be updated on time and it's especially  could be the case for events table. This is why we call this test event freshness anomalies. But although you update the table at the right cadence at the right time, the underlying data itself could still be in a delay because, for example, if you have an events table you could pull the events from the production systems and update the tables like every hour, every day, regularly, but the underlying event time would be delayed because you have some queue in the production system and the events are accumulating and the depth there is growing and then you get events, but you get them with a delay.

So the volume could still be fine, the updates could still be fine, but the underlying event time could be with the delay, so if you will provide a timestamp column, we will see usually we get events with event time up to yesterday, and all of a sudden now it's up to the day before yesterday, so we will detect anomalies normally using that.

But let's use another example. If you want to walk with like you have a fixed rigid SLAs that you want to enforce on your sources, and you don't want to use anomaly detection, and you still want to detect that your event table contains data up to yesterday, you can leverage the rest of the dbt test that exists in the ecosystem.

For example, I can use the dbt utils recency test. You can see that we have a small IDE with a prebuilt template that we provide out of the box, so you don't need to think what are the required parameters. We also say if the parameter is optional, so we can remove them. If, for example, you will remove a parameter that is mandatory, we will give you an error.