Tech Lead Data Engineer

AgileEngine; Mexico; Hybrid; PHP 9,124,000 - 12,774,000; Full time

Benefits offered by this job

Professional growth

Competitive USD-based compensation

Exciting projects

Flextime

Job summary

AgileEngine seeks a Tech Lead Data Engineer to own data pipelines and the analytical architecture for a large-volume marketing analytics platform. You will define partitioning strategies, file formats, and schema, while steering near-real-time processing for OLAP workloads on an S3-backed data lake.

You will design and govern ETL pipelines, define DAG-based orchestration with Airflow, drive AWS data stack including Athena and EKS, and lead a team of senior developers while upholding code quality

Qualifications

• 7+ years of engineering experience designing and implementing ETL pipelines for large-volume data systems.
• Hands-on OLAP-style data architecture with serverless query engines; Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, or similar.

Responsibilities

• Design and own ETL pipelines that extract, transform, and validate data at scale.
• Make architectural decisions on partitioning, file formats, schema, and near-real-time processing for OLAP workloads.
• Own the design of scheduled batch workflows (DAGs) on the client’s Airflow setup, defining pipeline structure and triggering strategy.
• Drive use of the client's AWS data stack (S3, Athena, EKS) and collaborate with DevOps on infra requirements.
• Review PRs and enforce code quality.
• Guide senior developers and align with client engineering practices.

Skills

7+ years of engineering experience

OLAP-style architecture

Athena

Trino/Presto

BigQuery

Snowflake

Spark SQL

ClickHouse

Parquet/ORC

Python

FastAPI/Flask

REST

GraphQL

Docker

PostgreSQL

Mac/Linux terminal

Airflow

Dagster/Prefect/Luigi/Step Functions

AWS data stack (S3, Athena, EKS)

Airflow architecture leadership

AI-assisted development tools (Claude)

Tools

Airflow

Dagster

Prefect

Luigi

Step Functions

Job description

We are looking for a Tech Lead Data Engineer to own the data pipeline and analytical architecture layer for a large-volume marketing analytics platform. You will make architectural decisions around partitioning strategy, file formats, schema design, and near-real-time processing for OLAP-oriented workloads built on an S3-backed data lake. You will design and govern ETL pipelines, define DAG-based orchestration strategies using Airflow, drive the AWS data stack including Athena and EKS, and lead a team of senior developers while enforcing code quality standards.

What you will do

• Design and own ETL pipelines that extract, transform, and validate data from internal databases and external APIs at scale.
• Make architectural decisions on partitioning strategy, file formats, schema and data-type strategy, and near-real-time processing for large-volume, OLAP-oriented data systems built on an object-storage data lake.
• Own the design of scheduled batch workflows (DAGs) on the client's Airflow setup, defining pipeline structure, dependencies, and triggering strategy, while driving architectural discussions. Not responsible for administering Airflow itself.
• Drive use of the client's AWS data stack (S3-backed data lake, Athena, EKS/Kubernetes), and partner directly with the client's DevOps team to clarify functional and non-functional requirements.
• Review pull requests and enforce code quality standards.
• Guide senior developers and ensure alignment with the client's engineering practices.

Must haves

• 7+ years of engineering experience, with a proven track record designing and implementing ETL pipelines and making architectural decisions for large-volume data systems.
• Hands-on experience with OLAP-style analytical data architecture. Experience with Athena, Trino/Presto, BigQuery, Snowflake, Spark SQL, ClickHouse, or similar technologies is acceptable; a specific stack isn't mandatory as long as the OLAP depth is real.
• Hands-on experience designing against a data lake sitting on object storage (S3 or equivalent) queried via a serverless engine — including partitioning strategy, file formats (Parquet/ORC), and the cost/performance tradeoffs that come with them. Athena specifically is a plus, not a requirement.
• Deep familiarity with DAG-style workflow definition and triggering. The client orchestrates most batch processing through Airflow, so this role needs either substantial prior Airflow experience they can draw on to drive architectural conversations, or enough depth in a comparable orchestrator (Dagster, Prefect, Luigi, Step Functions) to ramp on Airflow quickly and lead those conversations from day one. Managing the Airflow deployment itself is out of scope.
• Practical experience across the AWS data ecosystem, including S3-backed data lakes, serverless query engines such as Athena or equivalent, and EKS/Kubernetes, with the ability to drive infrastructure conversations with DevOps.
• Strong backend proficiency in Python, including FastAPI or Flask.
• Comfortable working with REST and GraphQL.
• Experience with Docker and PostgreSQL for the transactional and application layer.
• Highly comfortable working in Mac/Linux terminal-centric environments.
• Practical, hands-on use of AI-assisted development tools (e.g., Claude Code), paired with the critical judgment to challenge AI output when it compromises long-term maintainability — including the leadership presence to set the standard for how the team uses AI tooling responsibly (e.g., flagging risky AI-driven shortcuts during PR review).
• Strong soft skills: the ability to hold and defend a technical opinion — challenging a stakeholder's or a tool's proposed “quick fix” with sound reasoning in pursuit of a solution that scales and is maintainable long-term, while still being pragmatic enough to ship.

Nice to haves

• Direct production experience with Athena.
• Working knowledge of TypeScript and React to guide integrations and review frontend-adjacent pull requests.
• Production experience building AI features using AWS Bedrock, LangChain, Pydantic AI, or similar technologies.
• Experience with monorepo tooling such as Nx or modern package managers such as Poetry, UV, or Yarn.
• Experience with Redis, caching layers, or SageMaker.
• Experience with marketing data structures, campaign management APIs, or digital advertising metrics.

Perks and Benefits

• Professional growth

Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps

• Competitive compensation

We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities

• A selection of exciting projects

Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands

• Flextime

Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive.