Senior PySpark Data Engineer - AWS Data Pipelines

Highbrow LLC; Charlotte (NC); On-site; USD 110,000 - 160,000; Full time

Job summary

Highbrow LLC seeks a Senior Data Engineer to design and optimize large-scale data pipelines using PySpark and Python in a Charlotte-based environment. Collaboration with data scientists and analysts is essential to translate requirements into robust solutions.

You will write advanced SQL for ETL, troubleshoot data issues, and contribute to code reviews and data quality checks. Experience with AWS services (S3/EMR/Glue/Lambda) is preferred to build cloud-native workflows.

Qualifications

• Experience in a data engineering or backend development role.
• Proficiency with PySpark and Python for scalable data transformations.
• Strong SQL and performance tuning abilities.
• Experience with AWS services S3/Glue/EMR/Lambda and related tools.
• Solid OOP fundamentals and design patterns.

Responsibilities

• Design, develop, and optimise large-scale data pipelines using PySpark and Python.
• Work with data scientists and analysts to translate requirements into technical solutions.
• Write and optimize complex SQL queries for ETL processes.
• Troubleshoot data issues and ensure timely, accurate resolution.
• Collaborate in code reviews and data quality checks to improve pipelines.
• Leverage AWS services to build cloud-native data workflows.
• Participate in performance tuning of data jobs.

Skills

PySpark

Python

SQL

Data pipelines

AWS

OOP

Data quality

Collaboration

Kafka

Tools

S3

EMR

Glue

Lambda

Redshift

Job description

Highbrow LLC seeks a Senior Data Engineer to design and optimize large-scale data pipelines using PySpark and Python in a Charlotte-based environment. Collaboration with data scientists and analysts is essential to translate requirements into robust solutions.

You will write advanced SQL for ETL, troubleshoot data issues, and contribute to code reviews and data quality checks. Experience with AWS services (S3/EMR/Glue/Lambda) is preferred to build cloud-native workflows.