Ingeniero de Datos

PROGRAMMING.COM; Azcapotzalco; Sur place; MXN 520 000 - 760 000; Plein temps

Résumé du poste

PROGRAMMING.COM is seeking a Data Engineer to build and operate robust data pipelines using Databricks, Airflow, or Dagster. You will develop efficient ETL/ELT workflows in Python and SQL for batch and streaming workloads, and collaborate with ML teams to prepare high‑quality datasets for training and production use.

You will model and maintain Delta/Parquet/Iceberg data assets, implement orchestration and monitoring, and ensure data quality with validation and audit logging.

Qualifications

• 3–6 years of experience as a Data Engineer or ETL Developer in a production environment.
• Proficiency in Python and SQL; strong familiarity with Databricks, Spark, or equivalent big-data frameworks.
• Experience with workflow orchestration tools such as Airflow, Dagster, Luigi or Prefect.
• Deep understanding of data modeling, data warehousing, and distributed data processing.
• Knowledge of modern data lakehouse architectures (Delta, Parquet, Iceberg).
• Familiarity with CI/CD, GitHub Actions, and data pipeline testing frameworks.
• Comfort working in a cross-functional environment with ML, product, and analytics teams.

Responsabilités

• Build and operate robust data pipelines for ingestion, cleaning, and transformation using Databricks, Airflow, or Dagster.
• Develop efficient ETL/ELT workflows in Python and SQL to support both batch and streaming workloads.
• Collaborate with ML and AI teams to deliver high-quality datasets for training, evaluation, and production features.
• Model and maintain structured data assets (Delta, Parquet, Iceberg) for reliability, versioning, and lineage tracking.
• Implement orchestration and monitoring — schedule jobs, track dependencies, and automate recovery from failures.
• Ensure data quality and compliance through validation frameworks, schema enforcement, and audit logging.
• Contribute to data platform evolution — evaluate tools, standardize best practices, and improve developer experience.
• Support performance and cost optimization across compute, storage, and orchestration systems.

Connaissances

Python

SQL

Data Modeling

Big Data

CI/CD

GitHub Actions

Outils

Databricks

Airflow

Dagster

Spark

Description du poste

Role: Data Engineer

Location: Remote (Requires occasional travel)

Responsibilities

• Build and operate robust data pipelines for ingestion, cleaning, and transformation using Databricks, Airflow, or Dagster.
• Develop efficient ETL/ELT workflows in Python and SQL to support both batch and streaming workloads.
• Collaborate with ML and AI teams to deliver high-quality datasets for training, evaluation, and production features.
• Model and maintain structured data assets (Delta, Parquet, Iceberg) for reliability, versioning, and lineage tracking.
• Implement orchestration and monitoring — schedule jobs, track dependencies, and automate recovery from failures.
• Ensure data quality and compliance through validation frameworks, schema enforcement, and audit logging.
• Contribute to data platform evolution — evaluate tools, standardize best practices, and improve developer experience.
• Support performance and cost optimization across compute, storage, and orchestration systems.

Qualifications

• 3–6 years of experience as a Data Engineer or ETL Developer in a production environment.
• Proficiency in Python and SQL; strong familiarity with Databricks, Spark, or equivalent big-data frameworks.
• Experience with workflow orchestration tools such as Airflow, Dagster, Luigi or Prefect.
• Deep understanding of data modeling, data warehousing, and distributed data processing.
• Knowledge of modern data lakehouse architectures (Delta, Parquet, Iceberg).
• Familiarity with CI/CD, GitHub Actions, and data pipeline testing frameworks.
• Comfort working in a cross-functional environment with ML, product, and analytics teams.

Nice to Have

• Experience with sports, telemetry, or sensor data pipelines.
• Familiarity with streaming frameworks (Kafka, Spark Structured Streaming, Flink).
• General knowledge of American football, the NFL, and college football
• Background in data governance, lineage, and observability tools (Monte Carlo, Great Expectations, Unity Catalog,OpenLineage).
• Experience with cloud infrastructure (AWS, GCP, or Azure) and containerization (Docker, Kubernetes).
• Exposure to best practices in machine-learning model management and MLOps