dbt test: expect_column_values_to_be_increasing

The expect_column_values_to_be_increasing test from dbt-expectations verifies that the values in a specified column are either strictly increasing or non-decreasing based on the configured strictness level. This test is essential for ensuring the logical and chronological order in sequential data elements, such as financial transactions, datetime fields, or successively recorded metrics.

How it Works

The expect_column_values_to_be_increasing test assesses the ordered sequence of values within a specified column to ensure each succeeding entry is greater than or equal to the previous one (if strictly is set to false), or strictly greater (if strictly is set to true).

Steps and Conditions:

1. Configure Column Sorting: The test is executed on the column defined by sort_column, which is sorted before applying the test conditions.
2. Definition of Increasing:
   • Strictly Increasing (default): Each value in the sequence must be strictly greater than the one before it.
   • Non-decreasing: Setting the strictly parameter to false allows values to be equal or greater than the previous one.
3. Row Conditions: Optional row_condition allows filtering which rows are tested, ensuring the flexibility to exclude certain data records based on defined criteria.
4. Group By: Optionally, data can be grouped using the group_by parameter, allowing distinct tests within subgroups of the dataset.
5. Evaluation: The test processes the sorted and conditioned data to verify the defined increment condition is met across the dataset or within the groupings.
6. Outcome:
   • Pass: All data points or groups comply with the increment condition.
   • Fail: Any data point or group violates the increment rule, indicating an error or anomaly in data sequencing or input.

Example Usage: Fintech

In a Fintech environment, ensuring that financial transactions or event dates are recorded in an increasing or chronological order is crucial for accurate reporting and analysis.

Consider a scenario in a Fintech company where transaction_data table logs each transaction with timestamps:

tests:
  - dbt_expectations.expect_column_values_to_be_increasing:
      sort_column: transaction_datetime
      strictly: true
      group_by: [user_id]

This example applies the expect_column_values_to_be_increasing test on the transaction_datetime column, ensuring every transaction timestamp is strictly increasing within each user_id grouping. This arrangement ensures no user has transactions logged out of sequence, a critical measure for financial audits, user activity tracking, and temporal data integrity.