AWS Python Developer with Pyspark

Polarits; Newark (NJ); Hybrid; USD 140,000 - 180,000; Full time

Job summary

Polarits in New Jersey is seeking an experienced Python Developer with strong AWS and PySpark skills to join our data engineering team. This backfill role requires hands-on experience building scalable data pipelines, processing large datasets, and integrating with cloud environments.

The ideal candidate has 10+ years of software development experience using Python, plus extensive ETL and data pipeline experience and familiarity with distributed data processing frameworks.

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, or a related field.
• 10+ years of experience in software development with a strong focus on Python.
• Hands-on PySpark experience for distributed data processing and AWS services.

Responsibilities

• Design, develop, and maintain data pipelines and ETL workflows using Python, PySpark, and AWS.
• Build and optimize large-scale data processing and data transformation solutions.
• Integrate various data sources and ensure data quality, performance, and reliability.
• Collaborate with data engineers, analysts, and architects to deliver end-to-end data solutions.
• Implement best practices for code optimization, error handling, and data validation.
• Participate in code reviews, documentation, and deployment automation.
• Ensure adherence to data security and compliance standards.

Skills

Python

PySpark

AWS

Data pipelines

SQL

Distributed data processing

Education

Bachelor's degree in Computer Science, Data Engineering, or related field

Tools

Airflow

Databricks

Git

Docker

Kubernetes

Job description

Title: Python Developer with AWS & Pyspark (It’s backfill Role)

Location: New Jersey

Mode of work: Hybrid

Visa: GC, GC EAD, USC, H4 EAD only

Looking for Local candidates only with 10+years of experience and final round interview will be In-person interview

Job Summary

We are seeking an experienced PythonDeveloper with strong expertise in AWSandPySpark to join our data engineering team. The ideal candidate will have hands-on experience developing scalable data pipelines, processing large data sets, and integrating with cloud-based environments. This role requires excellent problem-solving skills and a strong understanding of distributed data processing frameworks.

Key Responsibilities

• Design, develop, and maintain data pipelinesandETL workflowsusingPython,PySpark, andAWS services.
• Build and optimize large-scale data processinganddata transformation solutions.
• Integrate various data sources and ensure data quality, performance, and reliability.
• Collaborate with data engineers, analysts, and architects to deliver end-to-end data solutions.
• Implement best practices for code optimization,error handling, anddata validation.
• Participate in code reviews, documentation, and deployment automation.
• Ensure adherence to data security and compliance standards.

Required Skills & Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, or a related field.
• 10+ years of experience in software development with a strong focus on Python.
• Hands-on experience with PySpark for distributed data processing.
• Solid understanding of AWS cloud services such as S3, Glue, Lambda, EMR, Redshift, and Athena.
• Strong experience in ETL development and data pipeline orchestration.
• Familiarity with SQL and relational/non-relational databases.
• Excellent analytical, debugging, and communication skills.

Preferred Skills

• Experience with Airflow,Databricks, or other workflow management tools.
• Knowledge of CI/CD pipelines and version control tools like Git.
• Exposure to data lakeordata warehouse architectures.
• Familiarity with DockerorKubernetes for deployment.