Senior Data Engineer, Hands-on Pipelines & Cloud

Pyramid Consulting, Inc; México; Hybrid; MXN 600,000 - 900,000; Full time

Job summary

Pyramid Consulting, Inc. is seeking an experienced data engineering contributor who can work hands-on with Python, data pipelines, and Airflow on AWS. You will build and maintain reusable Python packages, automate ETL jobs, and optimize SQL for data warehouses while implementing DBT tests and CI/CD pipelines.

The role requires extensive experience with S3, IAM, and cloud-based data storage, plus the ability to orchestrate complex workflows and monitor pipeline health in a scalable environment.

Qualifications

• Advanced Python concepts including structures, modules, classes, inheritance and multi-threading.
• Experience building reusable Python packages for internal or public use.
• Ability to automate ETL processes and schedule jobs with Airflow DAGs.
• Track and reprocess failed job records in pipelines.
• Coordinate orchestration of pipelines in sequence or parallel.
• Troubleshoot data pipeline errors and fix issues.
• Export/import data to/from CSV, JSON, XML, preferably from S3 or similar storage.
• Proficiency with AI IDE tools is a plus.
• Advanced SQL including joins, CTEs and subqueries; optimize queries for data warehouses.
• Testing and documentation: Python unit/integration/system tests; DBT data quality tests.
• Code generation using Python and Jinja templates.
• GitHub usage including CI/CD implementation from scratch.
• AWS data storage (S3), ECS, IAM, security best practices and DBT profiles access.

Responsibilities

• Develop reusable Python packages for internal or public usage.
• Produce automated ETL pipelines and manage Airflow DAGs.
• Monitor and reprocess failed pipeline runs.
• Orchestrate multiple pipelines to run in sequence or in parallel.
• Troubleshoot and fix data pipeline errors and performance issues.
• Export/import data to/from formats like CSV/JSON/XML from cloud storage.
• Implement DBT tests for data validation and quality checks.
• Create code using configurations via Python and Jinja templates.
• Implement CI/CD pipelines with GitHub for automated deployments.
• Integrate Airflow with AWS services and DBT models, triggering DBT jobs or data moves.

Skills

Python programming

Data pipelines

Airflow

ETL development

SQL mastery

CI/CD

GitHub

Tools

Airflow on AWS

DBT

GitHub CI/CD

S3 & AWS IAM

Redshift / EMR

Job description

Pyramid Consulting, Inc. is seeking an experienced data engineering contributor who can work hands-on with Python, data pipelines, and Airflow on AWS. You will build and maintain reusable Python packages, automate ETL jobs, and optimize SQL for data warehouses while implementing DBT tests and CI/CD pipelines.

The role requires extensive experience with S3, IAM, and cloud-based data storage, plus the ability to orchestrate complex workflows and monitor pipeline health in a scalable environment.