Lead AWS Data Engineer

Siri InfoSolutions Inc; Owings Mills (MD); On-site; USD 150,000 - 210,000; Full time

Job summary

Siri InfoSolutions Inc is seeking a senior data engineer to design and maintain data lakes, marts and meshes, leveraging Python, AWS Data Services, DBT and Airflow on a modern data stack.

You will collaborate with business and IT to define data models, migrations, and roadmaps while ensuring data quality and performance across complex pipelines.

Qualifications

• 12+ years of hands-on experience with Python, AWS Data Services, DBT, Airflow (Astronomer), SQL and PySpark.
• Very good SQL and Data Warehousing lifecycle knowledge mandatory.
• Experience in building data pipelines and orchestration using DBT and Apache Airflow.

Responsibilities

• Lead the design of reliable, secure data lakes, data marts and data meshes.
• Develop and maintain data integration solutions using Python and AWS Data Services with DBT and Airflow.
• Collaborate with stakeholders to define data strategy, models, and migration roadmaps.
• Translate complex technical constraints into business insights and manage expectations across teams.
• Develop, modify and debug existing data pipelines per business needs.
• Troubleshoot, tune and optimize code for better performance.
• Manage new requirements, review existing jobs, and fix performance issues.
• Guide and upskill junior/mid-level data engineers on best practices.
• Document all data flows, mappings, sessions and workflows.
• Handle tickets and problem ticket analysis in Agile/POD setups.

Skills

Python

SQL

Data Warehousing

ETL

Data Modeling

PySpark

Airflow

Tools

Snowflake

DB2

Postgres

DBT

Airflow

Job description

Must Have Technical/Functional Skills

• 12+ years of solid hands-on experience in Python, AWS Data Services, DBT, Apache Airflow (on Astronomer platform), SQL and PySpark
• Very Good hands-on knowledge on SQL and Data Warehousing life cycle is an absolute requirement.
• Experience in creating data pipelines and orchestrating using DBT & Apache Airflow
• Significant experience with data migrations and development of Operational Data Stores, Enterprise Data Warehouses, Data Lake and Data Marts.
• Experiencing in fixing performance issues / parallelism, Data model exposure is a must.
• Good to have: Experience with cloud ETL and ELT in one of the tools like Glue/EMR or any other ELT tool
• Experience in using Snowflake, DB2, Postgres and other database technologies is plus.
• Excellent communication skills to liaise with Business & IT stakeholders.
• Expertise in planning execution of a project and efforts estimation.
• Exposure to working in Agile ways of working

Roles & Responsibilities

• Lead the design of reliable, secure, and highly available data lakes, data marts and data meshes
• leads the design, development, and maintenance of data integration solutions using Python, AWS Data Services, DBT ensuring data pipeline & data quality
• Collaborate with business stakeholders and architects to define the enterprise data strategy, data models, and migration Roadmap
• Translate complex technical constraints into business insights and manage expectations across cross-functional teams
• Develop, modify, configure & debug existing data pipeline as per the business requirement.
• Troubleshoot and resolve technical issues. Debug, tune and optimize code for optimal performance
• Manage the new requirements, Review the existing jobs, Perform gap analysis & Fixing performance issues, etc.
• Guide, coach, and upskill junior and mid-level data engineers on best practices, coding standards, and modern data patterns
• Document all data flow & mappings, sessions and workflows
• Ticket handling and problem ticket analysis skills in Agile /POD approach