Build data pipelines: batch ETL with Apache Airflow, real-time streaming with Kafka, data validation, monitoring, and quality checks.
When to use this skill
- Building an ETL pipeline
- Setting up Kafka consumers
- Validating data quality
Core concepts
This skill provides guidance on data pipelines best practices, patterns, and common pitfalls. It is designed to be loaded on demand when a relevant task is detected.
Installation
curl -LO https://opencode-skills.example/downloads/data/data-pipelines.zip
unzip data-pipelines.zip -d ~/.config/opencode/skills/Restart OpenCode — the skill loads automatically.
When it triggers
- building an ETL pipeline
- setting up Kafka consumers
- validating data quality