ETL Data Test and Validate
Data Warehouse, Snowflake, 2024
Setup
Installed DBT Core locally. The install configuration can be described by the command: dbt debug
Installed DBT Utils
Basic Test
These are the basic must-do test to validate data. Here are the common built-in tests you can use:
Unique Test: Ensures that each value in the column is unique.
- name: column_name
tests:
- unique
Not Null Test: Ensures that no values in the column are null.
- name: column_name
tests:
- not_null
Accepted Values Test: Ensures that each value in the column is one of an allowed set of values.
- name: column_name
tests:
- accepted_values:
values: ['value1', 'value2', 'value3']
Relationships Test: Ensures referential integrity between two tables, similar to a foreign key constraint.
- name: column_name
tests:
- relationships:
to: ref('other_table')
field: other_column
Expression is True Test: Ensures that a SQL expression is true for all rows.
- name: column_name
tests:
- expression_is_true:
expression: "column_name > 0"