ETL Data Test and Validate

Data Warehouse, Snowflake, 2024

Setup

  1. Installed DBT Core locally. The install configuration can be described by the command: dbt debug image

  2. Installed DBT Utils image

    image

Basic Test

These are the basic must-do test to validate data. Here are the common built-in tests you can use:

Unique Test: Ensures that each value in the column is unique.

- name: column_name
  tests:
    - unique

Not Null Test: Ensures that no values in the column are null.

- name: column_name
  tests:
    - not_null

Accepted Values Test: Ensures that each value in the column is one of an allowed set of values.

- name: column_name
  tests:
      - accepted_values:
          values: ['value1', 'value2', 'value3']

Relationships Test: Ensures referential integrity between two tables, similar to a foreign key constraint.

- name: column_name
  tests:
    - relationships:
        to: ref('other_table')
        field: other_column

Expression is True Test: Ensures that a SQL expression is true for all rows.

- name: column_name
  tests:
    - expression_is_true:
        expression: "column_name > 0"