Lead Data Engineer

AgileEngine, LLC; Brasil; Presencial; BRL 260 000 - 520 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

100% remote

Flexible hours

Annual learning budget

Mentorship program

Tech Talks

Resumo da oferta

AgileEngine, LLC is seeking a Lead Data Engineer to own the data pipeline and analytical architecture for a high-volume marketing analytics platform. You will drive partitioning, schema design, and near-real-time processing for OLAP workloads on an S3-backed data lake, and steer ETL pipelines and Airflow DAGs.

You will lead the AWS data stack with Athena and EKS, mentor a team of senior developers, review PRs, ensure code quality, and collaborate with DevOps to align requirements with

Qualificações

• 7+ years of engineering experience with ETL pipelines and large-volume data systems.
• Experience with OLAP-style architectures and data lake design on object storage.

Responsabilidades

• Design and own ETL pipelines for data extraction, transformation, and validation at scale.
• Make architectural decisions on partitioning, file formats, and near-real-time processing.
• Own Airflow DAG design, scheduling, dependencies, and triggering strategies.
• Drive AWS data stack usage (S3 data lake, Athena, EKS/Kubernetes).
• Lead senior developers, review PRs, and enforce code quality.

Conhecimentos

ETL design

Data architecture

Leadership

Python

Airflow

Communication

AI governance

Ferramentas

Airflow

Dagster/Prefect

Athena/Trino/Presto

S3 Data Lake

Parquet/ORC

Spark SQL

EKS/Kubernetes

PostgreSQL

Docker

Descrição da oferta de emprego

We are looking for a Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural decisions around partitioning strategy, file formats, schema design, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. You will design and govern ETL pipelines, define DAG-based orchestration strategies using Airflow, drive the AWS data stack including Athena and EKS, and lead a team of senior developers while enforcing code quality standards.

What you will do

• Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale.
• Make architectural decisions around partitioning, file formats, schema and data-type strategy, and near-real-time processing for large-volume, OLAP-oriented data systems built on an object-storage data lake.
• Own the design of scheduled batch workflows (DAGs) on the client's Airflow setup, defining pipeline structure, dependencies, and triggering strategies, and driving architectural discussions around them.
• Drive the use of the client's AWS data stack, including an S3-backed data lake, Athena, and EKS/Kubernetes.
• Partner directly with the client's DevOps team to clarify functional and non-functional requirements.
• Review pull requests and enforce code quality standards.
• Guide senior developers and ensure alignment with the client's engineering practices.

Must haves

• 7+ years of engineering experience, with a proven track record designing and implementing ETL pipelines and making architectural decisions for large-volume data systems.
• Hands-on experience with OLAP-style analytical data architecture, with experience in Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar technologies.
• Hands-on experience designing data lakes backed by object storage such as S3 or equivalent, including partitioning strategies, file formats such as Parquet/ORC, and cost/performance tradeoffs.
• Deep familiarity with DAG-style workflow definition and triggering, with substantial experience in Airflow or comparable orchestrators such as Dagster, Prefect, Luigi, or Step Functions.
• Practical experience across the AWS data ecosystem, including S3-backed data lakes, serverless query engines such as Athena or equivalent, and EKS/Kubernetes.
• Strong backend proficiency in Python, with experience using FastAPI or Flask.
• Comfortable working with REST and GraphQL.
• Experience with Docker and PostgreSQL for transactional and application layers.
• Highly comfortable working in Mac/Linux terminal-centric environments.
• Practical, hands-on experience with AI-assisted development tools such as Claude Code, combined with the critical judgment to challenge AI-generated output when it compromises long-term maintainability.
• Leadership experience setting standards for responsible use of AI tooling, including identifying risky AI-driven shortcuts during code review.
• Strong communication and technical judgment, with the ability to defend technical decisions, challenge quick fixes with sound reasoning, and balance long-term maintainability with pragmatic delivery.

Nice to haves

• Direct production experience with Athena.
• Working knowledge of TypeScript and React, sufficient to guide integrations and review frontend-adjacent pull requests.
• Production experience building AI features using AWS Bedrock, LangChain, Pydantic AI, or similar technologies.
• Experience with monorepo tooling such as Nx or modern package managers such as Poetry, UV, or Yarn.
• Experience with Redis and caching layers or SageMaker.
• Experience with marketing data structures, campaign management APIs, or digital advertising metrics.
• Growth without limits: build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
• Competitive compensation: get recognition that reflects your skills and impact, with regular performance and compensation reviews
• Flexibility: work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
• Meaningful, modern projects: build impactful products using modern technologies alongside global teams and leading brands
• Collaborative culture: join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
• Well-being & support: access local well-being programs and people-focused support tailored to your location