Lead Data Engineer – AWS Data Platform & PySpark

NorthBay Solutions LLC; Karachi Division; Hybrid; PKR 2,500,000 - 4,200,000; Full time

Job summary

NorthBay Solutions LLC seeks a Lead Data Engineer (AWS Data Platform) with 5–8+ years of experience to design, develop, and optimize large-scale data platforms and ETL/ELT pipelines. The role involves hands-on work with PySpark, AWS Glue, EMR, and Redshift, delivering robust data warehousing solutions.

The candidate will collaborate with a UAE-based customer and distributed teams, ensuring performance tuning, data quality, security, and governance. Hybrid work in Pakistan is offered.

Qualifications

• 5 to 8+ years of experience in Data Engineering and Data Warehousing.
• Strong hands-on experience with PySpark.
• Extensive experience with AWS Glue.
• Experience building and managing workloads on Amazon EMR.
• Strong expertise in Amazon Redshift.
• Excellent SQL development and query optimization skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark, AWS Glue, and Amazon EMR.
• Build and optimize data ingestion, transformation, and processing frameworks for structured and semi-structured data.
• Develop and maintain enterprise data warehouse solutions using Amazon Redshift.
• Write complex SQL queries, stored procedures, and data transformations to support analytics and reporting requirements.
• Implement ETL/ELT processes to move data efficiently across multiple systems and platforms.
• Perform performance tuning and optimization of Spark jobs, ETL pipelines, SQL queries, and Redshift workloads.
• Ensure data quality, integrity, security, and governance across data platforms.
• Troubleshoot production issues and perform root cause analysis for data-related incidents.

Skills

PySpark

AWS Glue

Amazon EMR

SQL

Data warehousing

Performance tuning

Data governance

Distributed processing

Troubleshooting

Education

Bachelor's degree in CS/SE/IT

Tools

Amazon Redshift

S3

IAM

CloudWatch

Job description

NorthBay Solutions LLC seeks a Lead Data Engineer (AWS Data Platform) with 5–8+ years of experience to design, develop, and optimize large-scale data platforms and ETL/ELT pipelines. The role involves hands-on work with PySpark, AWS Glue, EMR, and Redshift, delivering robust data warehousing solutions.

The candidate will collaborate with a UAE-based customer and distributed teams, ensuring performance tuning, data quality, security, and governance. Hybrid work in Pakistan is offered.