Data Engineer

Optomi; Jersey City (NJ); On-site; USD 150,000 - 190,000; Full time

Job summary

Optomi in partnership with a leading professional services firm seeks a Senior Data Engineer to design and implement scalable data pipelines in AWS using Python and PySpark. You will model data with SCD Type 1/Type 2, develop API-driven components with FastAPI or Flask, and deploy cloud-native data platforms across CI/CD pipelines.

Collaborate with cross-functional teams to ensure reliable, scalable, and secure data architecture, optimize SQL queries for enterprise reporting, and leverage

Qualifications

• 6+ years of Data Engineering experience.
• Strong experience with AWS services (ECS, Lambda, SNS, DynamoDB).
• Advanced Python programming skills.
• Hands-on PySpark and distributed data processing.
• Strong SQL development and optimization.
• Deep understanding of data modeling (SCD Type 1/Type 2).
• Experience building APIs with FastAPI, Flask, or similar Python frameworks.
• Experience implementing CI/CD and deployment automation.
• Experience using AI-assisted development tools such as GitHub Copilot.
• Strong troubleshooting, analytical, and problem-solving abilities.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
• Investigate, troubleshoot, and optimize complex data pipelines.
• Develop and optimize SQL queries supporting enterprise reporting and analytics.
• Design and implement data models utilizing SCD Type 1 and Type 2 methodologies.
• Build, deploy, and support APIs within AWS environments.
• Develop solutions utilizing AWS services such as ECS, Lambda, SNS, and DynamoDB.
• Implement and maintain CI/CD pipelines to support automated deployments.
• Utilize AI-powered development tools to enhance development efficiency and code quality.
• Collaborate with cross-functional engineering teams to deliver scalable cloud-based solutions.
• Monitor, support, and continuously improve platform performance and reliability.

Skills

AWS

Python

PySpark

SQL

ETL/ELT

Data modeling (SCD Type 1 & Type 2)

API development (FastAPI/Flask)

CI/CD

GitHub Copilot

Troubleshooting

Tools

AWS ECS

AWS Lambda

AWS SNS

DynamoDB

Job description

Optomi in partnership with a leading professional services firm, is seeking a highly skilled Senior Data Engineer with deep expertise in AWS, Python, PySpark, and SQL. The ideal candidate will have experience building scalable ETL/ELT solutions, designing data models utilizing SCD Type 1 and Type 2 methodologies, and developing API-driven solutions using Python frameworks such as Fast API or Flask. This engineer will work closely with cross-functional teams to design, deploy, and maintain cloud-native applications and data platforms while ensuring reliability, scalability, and operational excellence.

What the Right Candidate Will Enjoy:

• Building scalable cloud-native solutions within AWS.
• Working extensively with Python, PySpark, and modern data engineering technologies.
• Designing and deploying APIs that support enterprise applications.
• Leveraging AI-powered development tools such as GitHub Copilot.
• Collaborating with highly technical engineering and data teams.
• Solving complex data challenges within a modern cloud environment.

What Type of Experience Does the Right Candidate Have:

• 6+ years of Data Engineering experience.
• Strong experience with AWS services including ECS, Lambda, SNS, and DynamoDB.
• Advanced Python programming skills.
• Hands-on experience with PySpark and distributed data processing.
• Strong SQL development and optimization experience.
• Deep understanding of data modeling concepts, including SCD Type 1 and Type 2.
• Experience building and deploying APIs using FastAPI, Flask, or similar Python frameworks.
• Experience implementing CI/CD processes and deployment automation.
• Experience working with AI-assisted development tools such as GitHub Copilot.
• Strong troubleshooting, analytical, and problem-solving abilities.

What the Responsibilities Are of the Right Candidate:

• Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
• Investigate, troubleshoot, and optimize complex data pipelines.
• Develop and optimize SQL queries supporting enterprise reporting and analytics.
• Design and implement data models utilizing SCD Type 1 and Type 2 methodologies.
• Build, deploy, and support APIs within AWS environments.
• Develop solutions utilizing AWS services such as ECS, Lambda, SNS, and DynamoDB.
• Implement and maintain CI/CD pipelines to support automated deployments.
• Utilize AI-powered development tools to enhance development efficiency and code quality.
• Collaborate with cross-functional engineering teams to deliver scalable cloud-based solutions.
• Monitor, support, and continuously improve platform performance and reliability.