# Data Quality Frameworks

> Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

Source: https://skillsagentes.com/skills/wshobson/agents/data-quality-frameworks
Repository: https://github.com/wshobson/agents
Author: wshobson
License: MIT
Updated: hace 2 meses
Context cost: 49 tok installed, 1.1k tok once triggered, 4k tok with every bundled file
Bundle: 2 files, 16 KB
Permissions requested: none declared

## Install

```bash
npx -y skills add wshobson/agents --skill data-quality-frameworks --agent claude-code
```

## What it does

- Configura validación de calidad de datos con Great Expectations, incluyendo suites de expectativas por columna
- Define patrones de pruebas dbt y contratos de datos para pipelines
- Genera reportes de validación con estado pass/fail por tabla y detalle de checks fallidos
- Establece dimensiones de calidad de datos: completitud, unicidad, validez, precisión, consistencia y actualidad

## Use it when

- Implementar checks de calidad de datos en pipelines
- Configurar validación con Great Expectations
- Construir suites de pruebas dbt completas
- Establecer contratos de datos entre equipos

## What triggers it

- "Configura Great Expectations para validar la tabla orders"
- "Ayúdame a crear una suite de pruebas dbt para mis modelos"
- "Necesito establecer un contrato de datos entre equipos"

## Before you install

- Requiere instalar great_expectations vía pip y, opcionalmente, dbt configurado en el proyecto.
- Needs on PATH: pip

## Files

- SKILL.md — 4 KB
- references/details.md — 11 KB

## SKILL.md

Reproduced verbatim from wshobson/agents under MIT. This section is the upstream document and is in English.

# Data Quality Frameworks

Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.

## When to Use This Skill

- Implementing data quality checks in pipelines
- Setting up Great Expectations validation
- Building comprehensive dbt test suites
- Establishing data contracts between teams
- Monitoring data quality metrics
- Automating data validation in CI/CD

## Core Concepts

### 1. Data Quality Dimensions

| Dimension        | Description              | Example Check                                      |
| ---------------- | ------------------------ | -------------------------------------------------- |
| **Completeness** | No missing values        | `expect_column_values_to_not_be_null`              |
| **Uniqueness**   | No duplicates            | `expect_column_values_to_be_unique`                |
| **Validity**     | Values in expected range | `expect_column_values_to_be_in_set`                |
| **Accuracy**     | Data matches reality     | Cross-reference validation                         |
| **Consistency**  | No contradictions        | `expect_column_pair_values_A_to_be_greater_than_B` |
| **Timeliness**   | Data is recent           | `expect_column_max_to_be_between`                  |

### 2. Testing Pyramid for Data

```
          /\
         /  \     Integration Tests (cross-table)
        /────\
       /      \   Unit Tests (single column)
      /────────\
     /          \ Schema Tests (structure)
    /────────────\
```

## Quick Start

### Great Expectations Setup

```bash
# Install
pip install great_expectations

# Initialize project
great_expectations init

# Create datasource
great_expectations datasource new
```

```python
# great_expectations/checkpoints/daily_validation.yml
import great_expectations as gx

# Create context
context = gx.get_context()

# Create expectation suite
suite = context.add_expectation_suite("orders_suite")

# Add expectations
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
)

# Validate
results = context.run_checkpoint(checkpoint_name="daily_orders")
```

## Detailed patterns and worked examples

Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.

## Summary: {total_passed}/{total_tables} tables passed")
        report.append("")

        for table, result in results.items():
            status = "✅" if result.passed else "❌"
            report.append(f"### {status} {table}")
            report.append(f"- Expectations: {result.total_expectations}")
            report.append(f"- Failed: {result.failed_expectations}")

            if not result.passed:
                report.append("- Failed checks:")
                for detail in result.details:
                    if not detail["success"]:
                        report.append(f"  - {detail['expectation']}: {detail['observed_value']}")
            report.append("")

        return "\n".join(report)

# Usage
context = gx.get_context()
pipeline = DataQualityPipeline(context)

tables_to_validate = {
    "orders": "orders_suite",
    "customers": "customers_suite",
    "products": "products_suite",
}

results = pipeline.run_all(tables_to_validate)
report = pipeline.generate_report(results)

# Fail pipeline if any table failed
if not all(r.passed for r in results.values()):
    print(report)
    raise ValueError("Data quality checks failed!")
```

## Best Practices

### Do's

- **Test early** - Validate source data before transformations
- **Test incrementally** - Add tests as you find issues
- **Document expectations** - Clear descriptions for each test
- **Alert on failures** - Integrate with monitoring
- **Version contracts** - Track schema changes

### Don'ts

- **Don't test everything** - Focus on critical columns
- **Don't ignore warnings** - They often precede failures
- **Don't skip freshness** - Stale data is bad data
- **Don't hardcode thresholds** - Use dynamic baselines
- **Don't test in isolation** - Test relationships too

---

Skills Agentes — https://skillsagentes.com/skills/wshobson/agents/data-quality-frameworks