Senior Data Engineer, AWS Infrastructure Supply Chain Automation

Amazon Data Services, Inc.; Seattle (WA); Sur place; USD 140 000 - 210 000; Plein temps

Résumé du poste

Amazon Data Services, Inc. in Seattle seeks a Senior Data Engineer to grow our Data Lake and Data Warehouse systems using serverless AWS components. You will own ETL pipelines, data modeling, and analytics infrastructure to enable internal customers to answer critical business questions with data.

You will work with Python, Spark, SQL and AWS services to build scalable data solutions, maintain security controls, and document data assets for teams across AWS.

Qualifications

• 5+ years of data engineering experience.
• Experience with data modeling, warehousing and ETL pipelines.
• Experience with SQL.
• Experience in at least one modern scripting or programming language.

Responsabilités

• Develop and maintain automated ETL pipelines (with monitoring) using Python/Spark and AWS services like S3, Glue, Lambda, SNS, SQS.
• Implement and support reporting and analytics infrastructure for internal business customers.
• Develop and maintain data security and permissions for enterprise-scale data warehouse and data lake, including encryption and access controls.
• Develop data objects for business analytics using data modeling techniques.
• Develop and optimize data warehouse and data lake tables with partitioning and performance best practices.
• Develop and maintain metadata, data catalog, and user documentation for internal customers.
• Collaborate with internal business customers and software teams to gather requirements for data publishing and consumption.

Connaissances

Data engineering

ETL pipelines

SQL

Python/Scala/Java

Mentoring

Outils

Hadoop

Hive

Spark

EMR

Description du poste

AWS Infrastructure Services owns the design, planning, delivery, and operation of all AWS global infrastructure. In other words, we're the people who keep the cloud running. We support all AWS data centers and all of the servers, storage, networking, power, and cooling equipment that ensure our customers have continual access to the innovation they rely on. We work on the most challenging problems, with thousands of variables impacting the supply chain - and we're looking for talented people who want to help.

You'll join a diverse team of software, hardware, and network engineers, supply chain specialists, security experts, operations managers, and other vital roles. You'll collaborate with people across AWS to help us deliver the highest standards for safety and security while providing seemingly infinite capacity at the lowest possible cost for our customers. You'll experience an inclusive culture that welcomes bold ideas and empowers you to own them to completion.

We're looking for Senior Data Engineer to help us grow our Data Lake and Data Warehouse Systems, which is being built using a serverless architecture, with 100% native AWS components including Redshift Spectrum, Athena, S3, Lambda, Glue, EMR, Kinesis, SNS, CloudWatch and more! We own a world-class data lake that is used to drive multi-billion dollar decisions on a regular cadence and we're looking to improve on filling the lake quickly, with as little human intervention needed and democratize the data in the lake.

Our Data Engineers build the ETL and analytics solutions for our internal customers to answer questions with data and drive critical improvements for the business. Our Data Engineers use best practices in software engineering, data management, data storage, data compute, and distributed systems. We are passionate about solving business problems with data!

Key job responsibilities

• Develop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, Spark, SQL and AWS services such as S3, Glue, Lambda, SNS, SQS, KMS.
• Implement and support reporting and analytics infrastructure for internal business customers.
• Develop and maintain data security and permissions solutions for enterprise scale data warehouse and data lake implementations including data encryption and database user access controls and logging.
• Develop data objects for business analytics using data modeling techniques.
• Develop and optimize data warehouse and data lake tables using best practices for DDL, physical and logical tables, data partitioning, compression, and parallelization.
• Develop and maintain data warehouse and data lake metadata, data catalog, and user documentation for internal business customers.
• Work with internal business customers and software development teams to gather and document requirements for data publishing and data consumption via data warehouse, data lake, and analytics solutions.

About the team

About AWS

Diverse Experiences

AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn't followed a traditional path, or includes alternative experiences, don't let it stop you from applying.

Why AWS?

Amazon Web Services (AWS) is the world's most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating - that's why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses.

Inclusive Team Culture

Here at AWS, it's in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE), inspire us to never stop embracing our uniqueness.

Mentorship & Career Growth

We're continuously raising our performance bar as we strive to become Earth's Best Employer. That's why you'll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.

Work/Life Balance

We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there's nothing we can't achieve in the cloud.

Basic Qualifications:

• 5+ years of data engineering experience
• Experience with data modeling, warehousing and building ETL pipelines
• Experience with SQL
• Experience in at least one modern scripting or programming language, such as Python, Java, Scala, or NodeJS
• Experience mentoring team members on best practices

Preferred Qualifications:

• Experience with big data technologies such as: Hadoop, Hive, Spark, EMR
• Experience operating larg