AWS Data Eng

Turtle Trax S.A.; Región Centro; Teletrabalho; MXN 500 000 - 900 000; Tempo integral

Resumo da oferta

Turtle Trax S.A. is seeking a data pipeline engineer to design and maintain robust Python-based ETL processes in a remote setup for Mexico. The role focuses on building scalable data flows, integrating with cloud storage, and ensuring high data quality.

Responsibilities include developing reusable Python packages, orchestrating Airflow on AWS, and implementing tests with DBT. Strong SQL and cloud security awareness are expected.

Qualificações

• Advanced Python concepts including modules, classes and inheritance.
• Experience building reusable Python packages for internal/public use.
• Ability to automate ETL and schedule Airflow DAGs.
• Track and reprocess failed pipeline runs.
• Orchestrate pipelines to run sequentially or in parallel.
• Troubleshoot data pipeline issues and fixes.
• Export/import data in CSV/JSON/XML from cloud storage like S3.
• Experience with AI IDE tools.

Conhecimentos

Python concepts

Advanced Python

ETL automation

Airflow DAGs

Pipeline monitoring

Data export/import

AI IDE tool

Ferramentas

GitHub

CI/CD

Snowflake

AWS S3

Airflow

DBT

Descrição da oferta de emprego

Remote

Advanced english

Only México

Must have below:

Python and Data Pipelines:

• Advanced Proficiency in Python concepts like Code Structures, Modules, Packages, Class, SubClass, Inheritance, Multi-Threading and Functional Programming.
• Experience in developing reusable Python packages for internal or public usage
• Ability to write automating ETL processes and scheduling jobs like Airflow DAG.
• Ability to track job pipeline runs to reprocess error records
• Ability to orchestration different pipelines to run in sequence or parallel.
• Troubleshoot data pipeline errors and fix issues
• Export or Import data to/from various formats like CSV, JSON, XML etc preferably from S3 or other cloud storage.
• Experience in using AI IDE tool

SQL:

• Advanced SQL skills, including complex joins, CTE's and subqueries
• Experience in optimizing SQL queries for performance and optimization in data warehouse technologies preferably Snowflake

Testing and documentation:

• Proficiency in Python unit, integration and system test.
• Proficiency in implementing DBT tests for data validation and quality checks

Code Generation:

• Experience in generating code using configurations using python and jinja templates

Version control:

• Experience in GitHub, including implementing CI/CD process from scratch

AWS Expertise:

• Data Storage solutions:
• In depth understanding of AWS S3 for data storage, ECS, IAM including best practices for organization and security
• Cloud Security:
• Knowledge of AWS security best practices, including IAM roles, encryption standard, secure coding guidelines, DBT profiles access configurations and more.

Data Integration (nice to have):

• Experience with AWS lambda for serverless data processing tasks

Workflow Orchestration (nice to have):

• Proficiency in using Apache Airflow on AWS to design ,schedule and monitor complex data flows
• Ability to integrate Airflow with AWS services and DBT models such as triggering a DBT model or EMR or reading from s3 writing to redshift or snowflake. Experience in API integrations to Salesforce, SFTP, SharePoint, One Drive and others.

DBT Core/ Cloud Proficiency (nice to have):

• Experience in creating complex DBT models including full refresh, incremental models, snapshots, and documentation.
• Ability to write and maintain DBT macros for reusable code
• Experience in creating custom DBT macros using jinja and Python allowing for reusable components within DBT models

Monitoring and Logging (nice to have):

• Familiarity with AWS cloud watch, Data Dog for monitoring the pipelines and setting up alerts for workflow failures