Data Engineer

AgileEngine; Portugal; Hybrid; EUR 45,000 - 70,000; Full time

Benefits offered by this job

Professional growth

Competitive USD-based compensation

Flextime

Hybrid/remote work options

Projects with Fortune 500 clients

Job summary

AgileEngine is seeking a Data Engineer to lead large-scale migrations to Snowflake and optimize Airflow workflows within a dynamic data platform team. You will migrate about 1,000 datasets, adapt pipelines to Snowflake, and ensure data quality while coordinating across time zones, including Pacific Time constraints, to accelerate modernization by March 2027.

The role requires Python, SQL, Snowflake, Airflow, Kafka, Spark, Iceberg, Parquet, and AWS S3 experience, with a focus on reliability and

Qualifications

• 3+ years of Python development experience.
• Advanced SQL skills with complex joins and window functions.
• Snowflake data modeling expertise.
• Apache Airflow experience for development and testing.
• Iceberg and Parquet data formats proficiency.
• AWS S3 data lakes and lifecycle policies.
• Testing and debugging code effectively.
• 2+ years with large-scale Airflow/Spark data.
• Overlap until 12 PM Pacific Time.

Responsibilities

• Support a large-scale data platform modernization effort.
• Migrate approximately 1,000 datasets to Snowflake.
• Adapt pipelines to Snowflake following established architecture.
• Ensure data quality and reliability during migration.
• Support workflow scheduling and monitoring in Apache Airflow.
• Execute migrations including Airflow 2 to 3 and Iceberg move.
• Coordinate across time zones to accelerate modernization by March 2027.

Skills

Python

SQL

Data engineering fundamentals

Problem solving

Communication

Tools

Snowflake

Airflow

Kafka

Spark/PySpark

Iceberg

Parquet

AWS S3

Testing/debugging

Job description

We are looking for a Data Engineer to execute large-scale data platform migrations to Snowflake and support reliable Airflow workflows.

The mandatory requirements are 3+ years of Python, advanced SQL, Snowflake modeling, Airflow, Kafka, Spark/PySpark, Iceberg, Parquet, AWS S3, testing/debugging, 2+ years with large-scale Airflow/Spark data, overlap until 12 PM Pacific Time, and upper-intermediate English. Snowflake/Airflow migrations and AI tools are a plus.

What you will do

• Work embedded within the data engineering function, partnering closely with the data platform team to support a large-scale data platform modernization effort.
• Execute the migration of approximately 1,000 datasets to Snowflake.
• Adapt existing pipelines to Snowflake, following the core architecture and technical approach defined by the internal team.
• Ensure data quality and reliability throughout the migration process.
• Support workflow scheduling and monitoring in Apache Airflow 3.
• Execute time-sensitive migrations, including Airflow 2 to 3, Confluent region migration, and the move to Iceberg.
• Help accelerate the migration initiatives targeted for completion by the end of March 2027.

Must haves

• Strong software engineering fundamentals.
• 3+ years of hands-on Python development experience.
• Advanced SQL, including complex joins, CTEs, window functions, and performance tuning.
• Snowflake data modeling.
• Apache Airflow — developing and testing pipelines in Airflow and Python.
• Iceberg and Parquet data formats.
• AWS experience, particularly S3 (data lakes, partitioning, and lifecycle policies).
• Testing strategy experience, with the ability to hands-on debug and test code.
• At least 2 years of industry experience working with large-scale datasets, including Airflow and Spark.
• Overlap until 12 PM Pacific Time.

Nice to haves

• Hands-on experience with Snowflake migrations (data warehouse modernization, performance and cost optimization).
• Experience with Apache Airflow migrations (version upgrades, executor changes, or platform migrations).

Perks and Benefits

• Professional growth

Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps

• Competitive compensation

We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities

• A selection of exciting projects

Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands

• Flextime

Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive.