Lead Data Engineer, AWS Data Platform & Pipelines

NorthBay Solutions LLC; Islamabad; Hybrid; PKR 2,400,000 - 5,000,000; Full time

Job summary

NorthBay Solutions LLC is seeking a Lead Data Engineer to design, develop, and optimize large-scale data platforms and ETL/ELT pipelines. You will work hands-on with PySpark, AWS Glue, EMR, and Redshift to build scalable data pipelines and data warehouses.

The role requires collaborating with a UAE-based client and distributed teams, while aligning with UAE working hours. The ideal candidate has 5 to 8+ years of experience in data engineering, strong SQL skills, and a solid understanding of data

Qualifications

• 5 to 8+ years of experience in Data Engineering and Data Warehousing.
• Hands-on PySpark and AWS Glue expertise.
• Experience with EMR and Redshift workloads.
• Strong SQL development and optimization skills.
• Knowledge of data warehousing concepts and ETL/ELT processes.
• Experience tuning Spark jobs and data pipelines for performance.
• Experience handling large-scale distributed datasets.
• Strong debugging and analytical skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark, AWS Glue, and EMR.
• Build and optimize data ingestion, transformation, and processing for structured data.
• Develop and maintain enterprise data warehouse solutions using Redshift.
• Write complex SQL queries and data transformations to support analytics.
• Implement ETL/ELT processes to move data across systems efficiently.
• Perform performance tuning of Spark jobs, ETL pipelines, and SQL queries.
• Ensure data quality, integrity, security, and governance across platforms.
• Troubleshoot production issues and analyze root causes of data incidents.
• Collaborate with stakeholders and cross-functional teams on data requirements.
• Participate in code reviews and technical design discussions.
• Monitor pipelines and drive improvements and automation.
• Create and maintain technical documentation and data models.

Skills

PySpark

AWS Glue

Amazon EMR

Amazon Redshift

SQL

Data Warehousing concepts

ETL/ELT

Performance tuning

Distributed processing

Education

Bachelor's degree in Computer Science/Software Engineering/IT

Tools

AWS S3

IAM

CloudWatch

Lambda

Step Functions

Job description

NorthBay Solutions LLC is seeking a Lead Data Engineer to design, develop, and optimize large-scale data platforms and ETL/ELT pipelines. You will work hands-on with PySpark, AWS Glue, EMR, and Redshift to build scalable data pipelines and data warehouses.

The role requires collaborating with a UAE-based client and distributed teams, while aligning with UAE working hours. The ideal candidate has 5 to 8+ years of experience in data engineering, strong SQL skills, and a solid understanding of data