Data Engineer

Optomi; Jersey City (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

Optomi is seeking an experienced engineer with strong Python and cloud platform skills, focused on AWS, PySpark, and data modeling. The role involves building data pipelines, designing scalable APIs, and deploying services with CI/CD pipelines.

You will utilize AI-powered tools like GitHub Copilot to accelerate development and work with cross-functional teams to deliver robust cloud solutions. The ideal candidate has hands-on AWS experience (ECS, Lambda, SNS, DynamoDB), solid SQL skills with SCD

Qualifications

• Strong Python and PySpark skills with SQL/data modeling experience.
• Experience building data pipelines and data warehousing concepts (SCD Types 1 & 2).
• Proficient in API development on AWS (FastAPI/REST/Flask).
• Hands-on with CI/CD and GitHub Copilot for productivity.

Responsibilities

• Write efficient, maintainable Python/PySpark code for data processing.
• Troubleshoot data pipelines and optimize PySpark workloads.
• Develop and optimize SQL queries; implement SCD Type 1/2 in warehousing.
• Design, develop, and deploy APIs on AWS (ECS/Lambda).
• Utilize AWS services (ECS, Lambda, SNS, DynamoDB) for scalable apps.
• Build, test, and deploy apps using CI/CD pipelines.
• Leverage AI-powered tools like GitHub Copilot to improve code quality.
• Collaborate with cross-functional teams to deliver robust cloud solutions.
• Troubleshoot cloud infra and application performance issues.

Skills

Python

PySpark

SQL

Data modeling

SCD Type 1

SCD Type 2

API development

FastAPI

REST/RESTful/Flask

CI/CD

GitHub Copilot

AWS

ECS

Lambda

SNS

DynamoDB

Tools

AWS

Job description

Overview: We are seeking an experienced Engineer with strong programming and cloud platform skills. The ideal candidate will have hands-on experience with a variety of AWS services, a solid background in Python development, SQL, and data modeling, and the ability to leverage AI tools such as GitHub Copilot to enhance productivity.

Key Responsibilities:

• Write efficient, maintainable code in Python/Pyspark, leveraging frameworks like PySpark for data processing.
• Investigate and troubleshoot data pipelines using PySpark, with a strong understanding of data structures and processing.
• Develop and optimize SQL queries; understand and implement Slowly Changing Dimensions (SCD) Type 1 and Type 2 in data warehousing.
• Design, develop, and deploy APIs on AWS, ensuring scalability and reliability.
• Work extensively with AWS services such as ECS, Lambda, SNS, and DynamoDB.
• Build, test, and deploy applications using CI/CD pipelines, ensuring smooth and automated delivery.
• Utilize AI-powered development tools (e.g., GitHub Copilot) to accelerate coding and improve code quality.
• Collaborate with cross-functional teams to deliver robust cloud solutions.
• Troubleshoot and resolve issues related to cloud infrastructure and application performance.

Required Skills:

• Strong knowledge of the AWS platform and its core services (ECS, Lambda, SNS, DynamoDB).
• Proficient in Python programming; experience with PySpark is highly desirable.
• Solid understanding of SQL and data modeling concepts, including SCD Type 1 and Type 2.
• Experience in building and deploying APIs (FastAPI/REST/RESTful/Flask) on AWS /(ECS).
• Good understanding of CI/CD processes and tools.
• Ability to work with AI-powered development tools such as GitHub Copilot.