Senior Data Engineer - Data Lakes, Airflow, AWS | Remote

AgileEngine; New York (NY); Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design and implement scalable data lakes, warehouses, and lakehouse architectures for a high-volume financial data platform.

The role focuses on Python-based ETL/ELT pipelines, Airflow orchestration, and ingestion workflows from diverse third-party APIs, leveraging Snowflake, Spark, and AWS to deliver high-performance data infrastructure and governance.

Qualifications

• 5+ years of experience with Python in data engineering.
• Experience building data lakes, data warehouses, and lakehouses.
• Proficient in ETL/ELT design and modern orchestration with Airflow.
• Strong ability to translate business goals into technical roadmaps.

Responsibilities

• Design and implement Python data engineering solutions.
• Build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Develop robust ETL/ELT processes at scale using Python and Airflow.
• Create ingestion workflows from diverse 3rd party APIs.
• Optimize file formats (Parquet, Avro, ORC) for performance.
• Collaborate with AI/ML teams to support analytics initiatives.
• Act as a technical consultant to gather requirements and roadmaps.
• Work with Terraform to provision AWS and on-prem infrastructure.

Skills

Python

Data processing

Airflow

Communication

Data quality

Education

Bachelor's degree in computer science/engineering or equivalent

Tools

Airflow

Snowflake

Spark

AWS

GCP

Terraform

Job description

AgileEngine is seeking a Senior Data Engineer to design and implement scalable data lakes, warehouses, and lakehouse architectures for a high-volume financial data platform.

The role focuses on Python-based ETL/ELT pipelines, Airflow orchestration, and ingestion workflows from diverse third-party APIs, leveraging Snowflake, Spark, and AWS to deliver high-performance data infrastructure and governance.