Data Engineer - 1861

In All Media Inc; Teletrabalho; BRL 364.000 - 469.000; Tempo integral

Resumo da oferta

A nearshore managed service provider is seeking a Data Engineer to join a specialized data squad focused on migrating large datasets to Snowflake. The role involves adapting data pipelines, ensuring reliability, and managing cloud infrastructure, primarily using AWS services. Ideal candidates will have advanced SQL skills, hands-on experience with Snowflake and Python, and a commitment to best practices in data migration and testing. This position allows remote work within LATAM.

Qualificações

• Mastery of complex SQL joins, CTEs, and performance tuning.
• Hands-on experience with Snowflake data modeling.
• Strong Python skills for data pipeline development and testing.

Responsabilidades

• Execute migration of high-volume datasets to Snowflake.
• Modify and optimize data pipelines for Snowflake.
• Develop and monitor workflows in Apache Airflow.

Conhecimentos

Advanced SQL

Snowflake Proficiency

Python for Data

Big Data Tools

AWS Ecosystem

Testing Mindset

Ferramentas

Apache Airflow

Apache Spark

Apache Kafka

Descrição da oferta de emprego

Data Engineer

Position: Data Engineer

Location: Remote from LATAM

Contract Type: Full-time vendor

Time Zone Alignment: CT

About In All Media In All Media is a nearshore managed service provider focused on team augmentation and digital product delivery. We assemble senior, LATAM-based squads from our vetted Coderfull community (500+ engineers) that integrate seamlessly with client teams to deliver software, data, cloud, and AI initiatives with speed and rigor.

Project Overview

Join a specialized data squad dedicated to a large-scale modernization initiative. This project focuses on the migration of approximately 1,000 datasets to Snowflake, utilizing Apache Airflow for orchestration. While the core architecture and technical approach are pre-defined by the internal team, you will be responsible for the execution, adaptation of pipelines, and ensuring the reliability of the new data ecosystem.

Key Responsibilities

• Data Migration: Execute the migration of high-volume datasets to Snowflake, ensuring architectural alignment and data integrity.
• Pipeline Adaptation: Modify and optimize existing data pipelines to leverage Snowflake’s native capabilities and performance.
• Orchestration & Workflow: Develop, test, and monitor complex DAGs in Apache Airflow to ensure seamless scheduling.
• Data Reliability: Implement comprehensive testing strategies to validate data quality throughout the migration lifecycle.
• Cloud Infrastructure: Manage and interact with AWS services, specifically S3 data lakes, partitioning strategies, and lifecycle policies.

Must-Have Skills

• Advanced SQL: Mastery of complex joins, CTEs, window functions, and performance tuning for large-scale data sets.
• Snowflake Proficiency: Hands-on experience with Snowflake data modeling and warehouse management.
• Python for Data: Strong ability to develop and test pipelines using Python and Airflow.
• Big Data Tools: Practical experience with Apache Spark and Apache Kafka for data processing and streaming.
• AWS Ecosystem: Deep understanding of S3 (data lakes, partitioning, and storage optimization).
• Testing Mindset: Proven experience in designing and executing testing strategies for data pipelines.

Nice-to-Have Skills

• Experience in high-velocity migration projects (legacy systems to cloud).
• Familiarity with data governance and security best practices within Snowflake.
• Certifications in AWS or Snowflake.

Common Requirements

• Solid understanding of data structures, algorithms, and software design principles.
• Commitment to CI/CD best practices and documentation.
• Fluent English for daily technical collaboration and project reporting.
• Ability to work within an Agile environment and follow pre-defined technical roadmaps.