AWS Data Engineer

Globalchannelmanagement; San Francisco (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading technology firm in San Francisco is looking for an experienced AWS Data Engineer to design and maintain data pipelines using AWS services. This role requires strong skills in Python and Spark, along with expertise in ETL development and data quality assurance. The ideal candidate will have 7+ years of experience in AWS Cloud Computing and a solid understanding of CI/CD practices. Candidates will work in a collaborative environment, optimizing data workflows and ensuring data integrity.

Qualifications

• 7+ years of experience in Amazon Web Services (AWS) Cloud Computing.
• Strong background in designing and maintaining data pipelines.
• Experience with ETL processes and tools.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Implement and manage ETL processes using AWS Glue and EMR.
• Monitor and optimize data workflows.

Skills

AWS Cloud Computing

Python

Spark

ETL Development

AWS Glue

Airflow

Data Quality Assurance

Tools

AWS EMR

CloudWatch

Datadog

Redshift

Job description

Overview

AWS Data Engineer needs 7+ years of experience in Amazon Web Services (AWS) Cloud Computing. AWS Data Engineer requires:

Responsibilities

• Strong background in designing, developing, and maintaining scalable data pipelines for ingestion and processing using Python, Spark, and AWS services
• ETL Development: Implement and manage ETL processes using AWS Glue and EMR for both batch and streaming data
• Data Storage Solutions: Design and maintain storage solutions using Medallion Architecture in S3 and Redshift
• Workflow Orchestration: Monitor and optimize data workflows using Airflow or other orchestration tools
• Data Quality: Ensure accuracy, consistency, and integrity throughout the data lifecycle
• CI/CD: Implement and understand CI/CD practices for automated data pipeline deployment
• System Monitoring: Utilize CloudWatch and Datadog for performance monitoring and reliability
• Collaboration: Work closely with cross-functional teams to gather requirements and deliver business-driven data solutions
• Communication: Strong ability to communicate project updates and technical details to stakeholders