Senior Data Engineer: AWS, PySpark & API Pipelines

Optomi; Jersey City (NJ); On-site; USD 150,000 - 190,000; Full time

Job summary

Optomi in partnership with a leading professional services firm seeks a Senior Data Engineer to design and implement scalable data pipelines in AWS using Python and PySpark. You will model data with SCD Type 1/Type 2, develop API-driven components with FastAPI or Flask, and deploy cloud-native data platforms across CI/CD pipelines.

Collaborate with cross-functional teams to ensure reliable, scalable, and secure data architecture, optimize SQL queries for enterprise reporting, and leverage

Qualifications

• 6+ years of Data Engineering experience.
• Strong experience with AWS services (ECS, Lambda, SNS, DynamoDB).
• Advanced Python programming skills.
• Hands-on PySpark and distributed data processing.
• Strong SQL development and optimization.
• Deep understanding of data modeling (SCD Type 1/Type 2).
• Experience building APIs with FastAPI, Flask, or similar Python frameworks.
• Experience implementing CI/CD and deployment automation.
• Experience using AI-assisted development tools such as GitHub Copilot.
• Strong troubleshooting, analytical, and problem-solving abilities.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
• Investigate, troubleshoot, and optimize complex data pipelines.
• Develop and optimize SQL queries supporting enterprise reporting and analytics.
• Design and implement data models utilizing SCD Type 1 and Type 2 methodologies.
• Build, deploy, and support APIs within AWS environments.
• Develop solutions utilizing AWS services such as ECS, Lambda, SNS, and DynamoDB.
• Implement and maintain CI/CD pipelines to support automated deployments.
• Utilize AI-powered development tools to enhance development efficiency and code quality.
• Collaborate with cross-functional engineering teams to deliver scalable cloud-based solutions.
• Monitor, support, and continuously improve platform performance and reliability.

Skills

AWS

Python

PySpark

SQL

ETL/ELT

Data modeling (SCD Type 1 & Type 2)

API development (FastAPI/Flask)

CI/CD

GitHub Copilot

Troubleshooting

Tools

AWS ECS

AWS Lambda

AWS SNS

DynamoDB

Job description

Optomi in partnership with a leading professional services firm seeks a Senior Data Engineer to design and implement scalable data pipelines in AWS using Python and PySpark. You will model data with SCD Type 1/Type 2, develop API-driven components with FastAPI or Flask, and deploy cloud-native data platforms across CI/CD pipelines.

Collaborate with cross-functional teams to ensure reliable, scalable, and secure data architecture, optimize SQL queries for enterprise reporting, and leverage