Data Engineer, Prime Video Personalization and Discovery Data Platform

Amazon; New York (NY); On-site; USD 118,000 - 206,000; Full time

Job summary

An innovative company is seeking a skilled Data Engineer to join their Prime Video Personalization and Discovery team. This role involves designing and building robust data pipelines that handle vast amounts of data, ensuring high-quality customer-level data is accessible for real-time processing. You'll collaborate with diverse teams, including Product, Data Scientists, and Software Developers, to enhance the digital video experience. If you're passionate about leveraging data to drive personalization and innovation in a fast-paced environment, this opportunity is tailor-made for you. Join a dynamic team and contribute to shaping the future of entertainment.

Benefits

Medical Benefits

401(k) Plan

Paid Time Off

Employee Discounts

Flexible Work Hours

Career Development Programs

Qualifications

• 3+ years of data engineering experience with strong SQL skills.
• Experience in building ETL pipelines and data modeling.

Responsibilities

• Design and build big data pipelines using Spark/Flink.
• Oversee production operations and optimize data delivery.

Skills

Data Engineering

ETL Pipelines

Data Modeling

SQL

Python

Scala

Big Data

CI/CD

Education

Bachelor's Degree in Computer Science or related field

Tools

AWS Redshift

AWS S3

AWS Glue

AWS EMR

AWS Kinesis

AWS Lambda

Job description

Data Engineer, Prime Video Personalization and Discovery Data Platform

Job ID: 2926526 | Amazon.com Services LLC

Are you interested in shaping the future of entertainment? Prime Video's technology teams are creating best-in-class digital video experience.

As a Prime Video technologist, you’ll have end-to-end ownership of the product, user experience, design, and technology required to deliver state-of-the-art experiences for our customers. You’ll get to work on projects that are fast-paced, challenging, and varied. You’ll also be able to experiment with new possibilities, take risks, and collaborate with remarkable people.

We’ll look for you to bring your diverse perspectives, ideas, and skill-sets to make Prime Video even better for our customers. With global opportunities for talented technologists, you can decide where a career in Prime Video Tech takes you!
Prime Video is a premium streaming service that offers a vast collection of digital videos—all with the ease of customers finding what they love to watch in one place.

Prime Video Personalization and Discovery team is building a new Data Platform. The platform will streamline ML and engineering teams to access and leverage high-quality, consistent customer-level data for offline processing and in real-time adhering to committed SLAs. We will act as guardians and gatekeepers of data shielding our customers from data issues while collaborating with data sources to address problems at their root. We will own and maintain "golden" datasets — centralized, reliable data assets commonly used across teams to train ML models and deliver top-notch personalized storefront experience, that is unique for every PV end-customer.

We will own Data Registry; we ensure that datasets are enriched with comprehensive classification and metadata, making them easily discoverable and accessible, and enabling ML and engineering teams to focus on deriving value for end customers rather than resolving data inconsistencies. Our goal is to save time for science and engineering teams to work with customer-level data, avoid duplicated boiler-plate data transformations, quality inspections, save IMR costs and increase experimentation velocity.

Key job responsibilities

1. Design and build big data pipelines via Spark/Flink that can handle Petabytes of data per month.
2. Build resilient data pipelines with extensive unit tests/integration tests with CI/CD development lifecycle.
3. Participate in oncall rotation supporting a large number of production data pipelines.
4. Manage and orchestrate version migrations across the metadata, transform, and storage layers.
5. Oversee and continually improve production operations, including optimizing data delivery, re-designing infrastructure for greater scalability, code deployments, bug fixes and overall release management and coordination.
6. Work closely with Product teams, Data Scientists, Software developers and Business Intelligence Engineer to explore new data sources and deliver the data.
7. Able to read, write, and debug data processing and orchestration code written Python/Scala etc following best coding standards (e.g. version controlled, code reviewed, etc.).

About the team

Our vision is to build a resilient, centralized data platform that distributes and streamlines access to the high quality, easily-discoverable customer-level data for offline processing and in real-time, enabling PVPD science and engineering teams to accelerate the focus on ML and engineering innovations to enhance personalization of PV end customer experiences.

BASIC QUALIFICATIONS

- 3+ years of data engineering experience
- Experience with data modeling, warehousing and building ETL pipelines
- Experience with SQL

PREFERRED QUALIFICATIONS

- Experience with AWS technologies like Redshift, S3, AWS Glue, EMR, Kinesis, FireHose, Lambda, and IAM roles and permissions
- Experience building MPP data transforms that process multiple Tb per day.

Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit this link for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.

Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $118,900/year in our lowest geographic market up to $205,600/year in our highest geographic market. Pay is based on a number of factors including market location and may vary depending on job-related knowledge, skills, and experience. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, please visit this link. This position will remain posted until filled. Applicants should apply via our internal or external career site.