Data Infrastructure Engineer

OpenAI; San Francisco (CA); Hybrid; USD 210,000 - 405,000; Full time

Job summary

Join a forward-thinking company as a key player in building and maintaining cutting-edge data infrastructure. This role offers the opportunity to work with innovative technologies like Kubernetes and Apache Spark, ensuring systems are scalable, reliable, and secure. Collaborate with talented engineers to enhance productivity and drive new features in a dynamic environment. If you thrive on challenges and are eager to learn, this position is perfect for you. Embrace the chance to shape the future of technology while contributing to a mission that benefits all of humanity.

Qualifications

• 4+ years in data infrastructure engineering or infrastructure engineering.
• Strong interest in building scalable, reliable, secure systems.

Responsibilities

• Design and maintain data infrastructure systems ensuring scalability and security.
• Empower engineers with excellent data tooling and systems.

Skills

Data Infrastructure Engineering

Infrastructure Engineering

Kubernetes

Apache Spark

Python

Terraform

Kafka

Azure EventHub

Vector Databases

Tools

Terraform

Kubernetes

Apache Spark

Kafka

Clickhouse

Job description

About the Team

You'll join the team that's behind OpenAI's data infrastructure that powers critical engineering, product, alignment teams that are core to the work we do at OpenAI. The systems we support include our data warehouse, batch compute infrastructure, streaming infrastructure, data orchestration system, data lake, vector databases, critical integrations, and more.

About the Role

The Applied Data Platform team designs, builds, and operates the foundational data infrastructure that enables products and teams at OpenAI.

You are comfortable with work such as scaling Kubernetes services, OLAP systems, debugging Kafka consumer lag, diagnosing distributed kv store failures, designing a system to retrieve image vectors with low latency.

You are well versed with infrastructure tooling such as Terraform, worked with Kubernetes, and have the SRE skill sets.

This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees.

In this role, you will:

• Design, build, and maintain data infrastructure systems such as distributed compute, data orchestration, distributed storage, streaming infrastructure while ensuring scalability, reliability, and security

• Ensure our data platform can scale reliably to the next several orders of magnitude

• Accelerate company productivity by empowering your fellow engineers & teammates with excellent data tooling and systems, providing a best in case experience

• Bring new features and capabilities to the world by partnering with product engineers, trust & safety and other teams to build the technical foundations

• Like all other teams, we are responsible for the reliability of the systems we build. This includes an on-call rotation to respond to critical incidents as needed

You might thrive in this role if you:

• Have 4+ years in data infrastructure engineering OR

• Have 4+ years in infrastructure engineering with a strong interest in data

• Take pride in building and operating scalable, reliable, secure systems

• Are comfortable with ambiguity and rapid change

• Have a voracious and intrinsic desire to learn and fill in missing skills-and an equally strong talent for sharing learnings clearly and concisely with others

Some of the technologies you'll be working with include Apache Spark, Clickhouse, Python, Terraform, Kafka, Azure EventHub, Vector DBs.

About OpenAI

OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the full spectrum of humanity.

We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or any other legally protected status.

OpenAI Affirmative Action and Equal Employment Opportunity Policy Statement

For US Based Candidates: Pursuant to the San Francisco Fair Chance Ordinance, we will consider qualified applicants with arrest and conviction records.

We are committed to providing reasonable accommodations to applicants with disabilities, and requests can be made via this link.

OpenAI Global Applicant Privacy Policy

At OpenAI, we believe artificial intelligence has the potential to help people solve immense global challenges, and we want the upside of AI to be widely shared. Join us in shaping the future of technology.

Compensation Range: $210K - $405K