Data Engineer ID94415

AgileEngine; Ernakulam; Remote; INR 1,200,000 - 2,500,000; Full time

Benefits offered by this job

Remote work

Local presence in India

Competitive INR compensation

Exciting projects

Job summary

AgileEngine, LLC. is seeking a Data Engineer to drive large-scale data platform migrations to Snowflake and to support reliable Airflow workflows. The role requires 3+ years of Python, advanced SQL, Snowflake modeling, and hands-on Airflow, Spark/PySpark, and AWS S3 experience.

You will work on a modernization effort, migrate about 1,000 datasets, and ensure data quality while coordinating Airflow 3 adoption and Iceberg transitions. Remote work with local presence in India is supported.

Qualifications

• 3+ years of hands-on Python development experience.
• Advanced SQL with complex joins, CTEs, window functions, and performance tuning.
• Snowflake data modeling capability.
• Apache Airflow—developing and testing pipelines in Airflow and Python.
• Apache Kafka exposure.
• Apache Spark / PySpark hands-on experience.
• Iceberg and Parquet data formats.
• AWS S3 data lake experience (partitioning and lifecycle policies).
• Testing strategy experience with hands-on debugging/testing code.
• 2+ years of industry experience with large-scale datasets including Airflow and Spark.
• Overlap until 12 PM Pacific Time.
• Upper-intermediate English.

Responsibilities

• Work within the data engineering function to support a large-scale data platform modernization.
• Migrate approximately 1,000 datasets to Snowflake.
• Adapt existing pipelines to Snowflake following internal architecture.
• Ensure data quality and reliability during migration.
• Support workflow scheduling and monitoring in Apache Airflow 3.
• Execute time-sensitive migrations, including Airflow 2 to 3, Confluent region migration, and the move to Iceberg.
• Accelerate migration initiatives targeted for completion by end of March 2027.

Skills

Python development

Advanced SQL

Snowflake data modeling

Apache Airflow

Apache Kafka

Apache Spark / PySpark

Iceberg & Parquet

AWS S3

Testing strategy

Large-scale datasets

English (upper-intermediate)

Tools

Snowflake

Airflow

Spark

Kafka

Parquet

Job description

We are looking for a Data Engineer to execute large-scale data platform migrations to Snowflake and support reliable Airflow workflows.

The mandatory requirements are 3+ years of Python, advanced SQL, Snowflake data modeling, and hands‑on experience with Airflow, Spark/PySpark, and AWS S3.

WHAT YOU'LL GAIN

• Remote work — Work from where you feel most productive.
• Local presence in India — Work in a structured and compliant environment aligned with Indian regulations.
• Competitive compensation in INR — Receive compensation in INR, plus support for learning, education, and wellness.
• Exciting projects — Work with modern technologies for global clients and fast-growing companies.

MUST HAVES

• Strong software engineering fundamentals.
• 3+ years of hands‑on Python development experience.
• Advanced SQL, including complex joins, CTEs, window functions, and performance tuning.
• Snowflake data modeling.
• Apache Airflow — developing and testing pipelines in Airflow and Python.
• Apache Kafka.
• Apache Spark, including hands‑on PySpark experience.
• Iceberg and Parquet data formats.
• AWS experience, particularly S3 (data lakes, partitioning, and lifecycle policies).
• Testing strategy experience, with the ability to hands‑on debug and test code.
• At least 2 years of industry experience working with large-scale datasets, including Airflow and Spark.
• Overlap until 12 PM Pacific Time.
• Upper‑intermediate English level.

NICE TO HAVES

• Hands‑on experience with Snowflake migrations (data warehouse modernization, performance and cost optimization).
• Experience with Apache Airflow migrations (version upgrades, executor changes, or platform migrations).
• Experience using AI tools for development.

WHAT YOU WILL DO

• Work embedded within the data engineering function, partnering closely with the data platform team to support a large-scale data platform modernization effort.
• Execute the migration of approximately 1,000 datasets to Snowflake.
• Adapt existing pipelines to Snowflake, following the core architecture and technical approach defined by the internal team.
• Ensure data quality and reliability throughout the migration process.
• Support workflow scheduling and monitoring in Apache Airflow 3.
• Execute time‑sensitive migrations, including Airflow 2 to 3, Confluent region migration, and the move to Iceberg.
• Help accelerate the migration initiatives targeted for completion by the end of March 2027.