Senior AWS Data Engineer

Programmers.io; Malvern (Chester County); On-site; USD 120,000 - 160,000; Full time

Job summary

Programmers.io is seeking an experienced AWS Data Engineer to design and maintain scalable ETL pipelines in the AWS cloud. The role requires deep hands-on expertise with Lambda, Redshift, S3, Athena, Glue, and CI/CD practices, as well as strong Python and PySpark skills.

You will collaborate with cross-functional teams to translate data requirements into reliable, cost-efficient data architectures, monitor performance, and ensure security and compliance across data platforms.

Qualifications

• Experience with AWS services and data engineering in cloud environments.
• Strong Python and PySpark for data transformation and processing.
• Design and optimization of scalable ETL pipelines on AWS Redshift and S3.
• Knowledge of data security, privacy, and compliance frameworks.

Responsibilities

• Design, develop, and maintain robust ETL workflows to load data into Redshift.
• Use AWS Lambda for real-time data processing and automation.
• Work with S3, Athena, Glue to support data pipelines.
• Write Python and PySpark for data transformations and queries.
• Collaborate with cross-functional teams to define data requirements.
• Monitor performance, reliability, and cost of cloud infrastructure.
• Troubleshoot data pipelines to minimize downtime and ensure accuracy.
• Implement automation and monitoring tools to improve reliability.

Skills

AWS Lambda

Redshift

S3

Athena

Glue

Python

PySpark

ETL pipelines

CloudWatch

Data warehousing

CI/CD

CloudFormation

Git

Data security

Agile

Tools

CloudFormation

Git

CI/CD pipelines

Job description

Job Description:

• We are seeking an experienced AWS Data Engineer with expertise in ETL processes, AWS Lambda, Redshift, and proficiency in Python and PySpark.
• This role requires hands-on experience with cloud-based architecture, data engineering, and working with large-scale data processing systems. The ideal candidate will play a critical role in building and maintaining scalable, efficient, and secure ETL pipelines within the AWS ecosystem.

Key Responsibilities:

• Design, develop, and maintain robust ETL workflows to extract, transform, and load data from various sources to AWS Redshift and other cloud-based systems.
• Leverage AWS Lambda to build serverless computing solutions for real-time data processing and automation.
• Work with AWS data storage and processing services, including S3, Redshift, Athena, and Glue, to support data processing pipelines.
• Utilize Python and PySpark to create and optimize data transformation logic and analytical queries.
• Collaborate with cross-functional teams to understand data requirements and provide efficient data solutions.
• Monitor and optimize cloud-based infrastructure for performance, reliability, and cost efficiency.
• Troubleshoot and resolve data pipeline issues, ensuring minimal downtime and high data accuracy.
• Implement automation and monitoring tools to enhance reliability.

Required Skills and Qualifications:

• Proven experience working with AWS services, including Lambda, Redshift, S3, Athena, Glue, and CloudWatch.
• Strong knowledge and hands-on experience with Python and PySpark for data transformation and processing.
• Expertise in building and optimizing ETL pipelines in a cloud environment.
• In-depth understanding of relational databases, data warehousing, and performance tuning in Redshift.
• Strong problem-solving and troubleshooting skills.
• Familiarity with infrastructure as code tools like AWS CloudFormation.
• Experience with version control (e.g., Git) and CI/CD pipelines.
• Solid understanding of data security, data privacy, and compliance frameworks.
• Ability to work in an agile, fast-paced environment.