Lead Data Engineer ID71008

AgileEngine, LLC.; Región Centro; Presencial; MXN 900.000 - 1.200.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive pay

Remote work 100%

Challenging projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Lead Data Engineer to own the data pipeline and analytics architecture for a large-volume marketing analytics platform built on an S3-backed data lake. You will shape partitioning, file formats, and near-real-time processing for OLAP workloads.

You will design and govern ETL pipelines, define DAG-based orchestration with Airflow, drive the AWS data stack (Athena, EKS), and lead a team of senior developers while enforcing code quality standards.

Formación

• 7+ years of engineering experience with ETL pipelines and large-volume data systems.
• Hands-on OLAP-style analytical data architecture experience with tools like Athena/Trino/Presto/Snowflake.
• Experience with object-storage-backed data lakes (S3 or equivalent) and partitioning strategies.
• Deep familiarity with DAG-style workflow orchestration (Airflow or alternatives).
• AWS data stack exposure: S3, Athena, EKS, serverless queries.
• Backend Python development (FastAPI/Flask).
• Experience with REST and GraphQL APIs.
• Docker and PostgreSQL for transactional/application layers.
• Comfort in Mac/Linux terminal environments.
• Experience with AI-assisted development tools and responsible AI use.
• Ability to defend technical opinions and work through ambiguity.

Responsabilidades

• Design and own ETL pipelines at scale.
• Make architectural calls on partitioning, file formats, and near-real-time processing for OLAP workloads.
• Lead DAG design, dependencies, and triggering strategy in Airflow (not administering Airflow).
• Drive AWS data stack usage and collaborate with DevOps to define requirements.
• Review PRs and enforce code quality standards.
• Guide senior developers and align with engineering practices.

Conocimientos

7+ years of engineering experience

OLAP-style analytical data

Object-storage-backed data lakes

Airflow

AWS data stack

Python

REST/GraphQL

Docker

PostgreSQL

Mac/Linux

AI-assisted development tools

Ambiguity tolerance

Herramientas

Airflow

Athena

EKS/Kubernetes

Descripción del empleo

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.

WHY JOIN US

If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!

ABOUT THE ROLE

We are looking for a Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural decisions around partitioning strategy, file formats, schema design, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. You will design and govern ETL pipelines, define DAG-based orchestration strategies using Airflow, drive the AWS data stack including Athena and EKS, and lead a team of senior developers while enforcing code quality standards. The role requires a high degree of autonomy: you will often work on ad hoc or underspecified problems, defining the problem, gathering context, identifying constraints, and shaping the right technical approach before implementation.

WHAT YOU WILL DO

• Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale.
• Make architectural calls on partitioning, file formats, schema/data-type strategy, and near-real-time processing for large-volume, OLAP-oriented data systems built on an object-storage data lake.
• Own the design of scheduled batch workflows (DAGs) on the Airflow setup — defining pipeline structure, dependencies, and triggering strategy, and driving architectural conversations about them. Not responsible for administering Airflow itself.
• Drive use of the AWS data stack (S3-backed data lake, Athena, EKS/Kubernetes), and partner directly with the DevOps team to clarify functional and non-functional requirements.
• Review PRs and enforce code quality standards.
• Guide senior developers and ensure alignment with established engineering practices.

MUST HAVES

• 7+ years of engineering experience, with a proven track record designing and implementing ETL pipelines and making architectural decisions for large-volume data systems.
• Hands-on experience with OLAP-style analytical data architecture — comparable experience with Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar is acceptable; a specific stack isn't mandatory as long as the OLAP depth is real.
• Object-storage-backed data lakes: hands‑on experience designing against a data lake sitting on object storage (S3 or equivalent) queried via a serverless engine — including partitioning strategy, file formats (Parquet/ORC), and the cost/performance tradeoffs that come with them. Athena specifically is a plus, not a requirement.
• Task orchestration: Deep familiarity with DAG‑style workflow definition and triggering. Most batch processing is orchestrated through Airflow, so this role needs either substantial prior Airflow experience they can draw on to drive architectural conversations, or enough depth in a comparable orchestrator (Dagster, Prefect, Luigi, Step Functions) to ramp on Airflow quickly and lead those conversations from day one. Managing the Airflow deployment itself is out of scope.
• AWS Ecosystem: practical comfort across the AWS data stack — S3‑backed data lake, serverless query engines (Athena or equivalent), and EKS/Kubernetes — with the ability to drive infrastructure conversations with DevOps.
• Backend proficiency in Python (FastAPI or Flask).
• Comfortable with REST and GraphQL.
• Docker and PostgreSQL for the transactional/application layer.
• Highly comfortable in Mac/Linux terminal‑centric environments.
• Practical, hands‑on use of AI‑assisted development tools (e.g., Claude Code), paired with the critical judgment to challenge AI output when it compromises long‑term maintainability — including the leadership presence to set the standard for how the team uses AI tooling responsibly (e.g., flagging risky AI‑driven shortcuts during PR review).
• Strong soft skills: the ability to hold and defend a technical opinion — challenging a stakeholder's or a tool's proposed "quick fix" with sound reasoning in pursuit of a solution that scales and is maintainable long‑term, while still being pragmatic enough to ship.
• Comfort with ambiguity (mandatory): work is frequently ad hoc and underspecified. This role requires defining the problem — gathering context, identifying constraints, and framing the work — before solving it, rather than waiting for a specification. Experience limited to well‑specified work executed through agent workflows is not a fit.

NICE TO HAVES

• Direct production experience with Athena specifically.
• Working knowledge of TypeScript/React — enough to guide integration and review frontend‑adjacent PRs, even if not the primary focus.
• Production AI features using AWS Bedrock, LangChain, Pydantic AI, or similar.
• Monorepo tooling (Nx) or modern package managers (Poetry, UV, Yarn).
• Redis/caching layers, SageMaker.
• Experience with marketing data structures, campaign management APIs, or digital advertising metrics.

PERKS AND BENEFITS

• Growth without limits: build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
• Competitive compensation: get recognition that reflects your skills and impact, with regular performance and compensation reviews
• Flexibility: work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
• Meaningful, modern projects: build impactful products using modern technologies alongside global teams and leading brands
• Collaborative culture: join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
• Well-being & support: access local well‑being programs and people‑focused support tailored to your location