Senior Data Engineer: Data Lakes, Airflow & Snowflake
AgileEngine, LLC.
Get more replies from employers
Send a job-specific resume in minutes.
Benefits offered by this job
Job summary
AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses, processing vast financial datasets daily. You will implement Python ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.
The role combines hands-on engineering with consulting responsibilities and data architecture roadmaps.
Qualifications
- 5+ years of Python experience in data engineering.
- 5+ years with data processing and analytics libraries (Pandas/Polars/PySpark/DuckDB).
- 2+ years with Spark and Snowflake.
- Expert level with Airflow and data pipelines.
- Strong cloud experience (AWS, GCP) and SQL optimization.
Responsibilities
- Design and implement Python data engineering solutions.
- Build scalable Data Lakes, Data Warehouses, and Lakehouses.
- Develop ETL/ELT processes at scale using Python and Airflow.
- Ingest data from diverse 3rd party APIs and sources.
- Optimize file formats (Parquet/Avro/ORC) for performance.
- Collaborate with AI/ML teams to enable analytics and model workloads.
- Translate business goals into technical roadmaps and standards.
- Work with Terraform to provision AWS/on‑prem infrastructure.
Skills
Education
Tools
Job description
AgileEngine is seeking a Senior Data Engineer to design scalable data lakes, warehouses, and lakehouses, processing vast financial datasets daily. You will implement Python ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.
The role combines hands-on engineering with consulting responsibilities and data architecture roadmaps.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
