Senior Data Engineer - Data Lakehouse & AI Pipelines

AgileEngine; Baltimore (MD); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Professional growth

Competitive compensation

Exciting projects

Flextime

Job summary

AgileEngine is seeking a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse architectures supporting a thematic research platform that processes large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.

Qualifications

• Bachelor's degree in computer science/engineering or equivalent experience.
• 5+ years of experience with Python (strong, hands-on Python experience is a must).
• 5+ years of experience with data processing, manipulation, and analytics libraries like Pandas, Polars, PySpark or DuckDB.
• 2+ years of experience with Big Data technologies (Spark, Snowflake).
• Expert-level knowledge of pipeline orchestration using Airflow or similar industry-standard tools.
• Deep understanding of Medallion Architecture, columnar file formats, and diverse database technologies (SQL, NoSQL, and Lakehouse architectures).
• Proven ability to work with 3rd party APIs for complex data ingestion tasks.
• Proficiency with modern Cloud platforms (AWS, GCP, Snowflake) and advanced SQL optimization.
• Exceptional soft skills with a proven ability to gather requirements from leadership and collaborate effectively across cross-functional teams.
• Excellence in optimizing complex data pipelines and troubleshooting data latency or consistency issues in massive datasets.

Responsibilities

• Design and implement Python Data Engineering solutions.
• Design and build scalable Data Lakes, Data Warehouses, and Data Lakehouses.
• Design and implement robust ETL/ELT processes at scale using Python, incorporating modern pipeline orchestration tools like Airflow.
• Develop sophisticated ingestion workflows from diverse 3rd party APIs and data sources.
• Manage and optimize various file formats (Parquet, Avro, ORC) and columnar storage to ensure high-performance data retrieval.
• Work with AI development tools to support and accelerate ongoing development, machine learning initiatives and advanced analytics.
• Act as a technical consultant for stakeholders and leadership to gather requirements, understand business goals, and translate them into technical roadmaps.
• Work with Terraform and other tools to build AWS and on-prem infrastructure.

Skills

Python

Pandas

PySpark

DuckDB

Airflow

Spark

Snowflake

ETL/ELT

API ingestion

SQL

Education

Bachelor's degree in computer science/engineering

Equivalent experience

Tools

Terraform

OpenSearch

Elasticsearch

Job description

AgileEngine is seeking a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse architectures supporting a thematic research platform that processes large volumes of financial data daily.

You will implement Python-based ETL/ELT pipelines, orchestrate workflows with Airflow, develop ingestion workflows from third-party APIs, and work with Snowflake, Spark, and AWS to deliver high-performance data infrastructure.