Principal Engineer, Big Data Platform

Jobtailor; San Francisco (CA); On-site; USD 250,000 - 300,000; Full time

Job summary

Pinterest is seeking a senior leader to direct the data infrastructure strategy and the technical direction for big data and AI applications. You will oversee scalable frameworks, compute engines, and scheduling for petabyte-scale datasets, while collaborating with internal customers on critical business use cases.

In this role, you’ll drive thought leadership on fast and reliable data processing and storage at scale, contribute to the long-term roadmap, and mentor cross-functional teams to

Qualifications

• 12+ years of industry experience with a proven track record of technical excellence.
• 8+ years building and supporting large scalable Kubernetes or big data platforms.
• Deep knowledge of big data / ML technologies (Flink, Spark, Presto, Kubernetes, Ray, PyTorch/TensorFlow).

Responsibilities

• Lead the strategy and technical direction of Pinterest’s data infrastructure for big data and AI applications.
• Build and scale data infrastructure frameworks for petabyte-scale datasets, including compute engines, job management, resource management, scheduling, and remote shuffling.
• Work with internal customers on critical business use cases relying on big data.
• Provide company-wide thought leadership on processing and storing data reliably, quickly, and efficiently at scale.
• Contribute to the team’s technical vision and long-term roadmap.

Skills

Big Data Technologies

Kubernetes Platform Development

Data Infrastructure Frameworks

Programming in Java, Go, Scala, or Py3

AWS Technologies

Education

Bachelor’s degree in Computer Science or related field

Tools

AWS

Job description

• Lead the strategy and technical direction of Pinterest’s data infrastructure for big data and AI applications
• Build and scale data infrastructure frameworks for petabyte-scale datasets, including compute engines, job management, resource management, scheduling, and remote shuffling
• Work with internal customers on critical business use cases relying on big data
• Provide company-wide thought leadership on processing and storing data reliably, quickly, and efficiently at scale
• Contribute to the team’s technical vision and long-term roadmap

Requirements

• 12+ years of industry experience with a proven track record of technical excellence
• 8+ years of experience building and supporting large scalable Kubernetes or big data platforms
• Deep knowledge of big data / ML technologies, including Flink, Spark, Presto, Kubernetes, Ray, PyTorch/TensorFlow
• Proficiency in one or more programming languages: Java, Go, Scala, or Python
• Experience with Kubernetes and AWS technologies
• Exceptional collaboration skills with cross-functional partners, ability to navigate ambiguity, make tradeoffs, and align stakeholders
• Bachelor’s degree in Computer Science, a related technical field, or equivalent experience

Core Competencies

Demonstrates expertise in leading the strategy and technical direction of data infrastructure for big data and AI applications, with a strong focus on building scalable frameworks and collaborating with cross-functional teams.

Highest-signal resume keywords

• Big Data Technologies
• Kubernetes Platform Development
• Data Infrastructure Frameworks
• Programming in Java, Go, Scala, or Python
• AWS Technologies

Hard Skills

• Big Data
• Machine Learning
• Flink
• Spark
• Presto
• Ray
• PyTorch
• TensorFlow
• Kubernetes
• Data Processing

Soft Skills

• Collaboration
• Stakeholder Alignment
• Navigating Ambiguity
• Tradeoff Decision-Making

Industry Keywords

• Data Infrastructure
• Petabyte-Scale Datasets
• Compute Engines
• Job Management
• Resource Management
• Scheduling
• Remote Shuffling

Tools & Technologies

• AWS