Senior Data Engineer: Data Lakes & Airflow Pipelines

AgileEngine; San Francisco (CA); Hybrid; USD 140,000 - 220,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouse architectures that support a thematic research platform handling large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. This role blends hands-on engineering with technical consulting responsibilities.

Qualifications

• 5+ years of Python experience.
• 5+ years data processing with Pandas, Polars, PySpark or DuckDB.
• 2+ years Big Data tech (Spark, Snowflake).
• Expert-level Airflow or similar.
• Medallion Architecture and columnar formats.
• Experience with 3rd party APIs for data ingestion tasks.
• Cloud platforms AWS/GCP/Snowflake and SQL optimization.
• Strong communication and cross-functional collaboration.
• Authorized to work for ANY employer in the US.
• Bachelor’s degree or equivalent.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Build Data Lakes, Data Warehouses, and Data Lakehouses.
• Create ETL/ELT pipelines with Python and Airflow.
• Develop ingestion workflows from third-party APIs.
• Manage Parquet/Avro/ORC formats for performance.
• Collaborate with AI teams to support ML initiatives.
• Translate business goals into technical roadmaps.
• Work with Terraform to build AWS and on-prem infrastructure.

Skills

Python

Data processing

SQL

Communication

Team collaboration

Education

Bachelor's degree

Tools

Airflow

Snowflake

Spark

Terraform

AWS

GCP

Job description

AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouse architectures that support a thematic research platform handling large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure. This role blends hands-on engineering with technical consulting responsibilities.