Senior Data Engineer: Scalable Pipelines & Data Platforms

Rad AI; San Francisco (CA); On-site; USD 145,000 - 190,000; Full time

Benefits offered by this job

Medical, Dental, Vision & Life

HSA with employer match

401(k)

Paid holidays

Flexible PTO

Annual offsite

Equipment stipend

Job summary

Rad AI is seeking a Senior Data Engineer in San Francisco to design and maintain scalable data pipelines, data architecture, and data products. You will collaborate with researchers to align data needs with model training and production monitoring.

You will own key projects, ensure data quality and security, and continuously improve the data infrastructure with modern tools and workflows.

Qualifications

• 5+ years of experience in data engineering.
• Experience designing distributed data pipelines using big data technologies.
• Hands-on experience building reliable, scalable data infrastructure and products.
• Strong experience with AWS data services (EMR, Batch).
• Proficiency with Spark and large-scale data processing.
• Experience with Metaflow for workflow orchestration.
• Knowledge of SQL and NoSQL databases (PostgreSQL, DynamoDB, Elasticsearch).
• Containerization with Docker and orchestration with Kubernetes.

Responsibilities

• Design and implement scalable data architecture and pipelines.
• Develop standards for style, maintenance, and best practices for a high-scale data platform.
• Collaborate with researchers and stakeholders to meet data needs including model training and monitoring.
• Own key data engineering projects from design to deployment.
• Ensure data quality, integrity, and security with validation, monitoring, and access controls.
• Evaluate data technologies and tools to improve efficiency and effectiveness.
• Monitor, maintain, and improve performance of data infrastructure.

Skills

Data pipelines

Spark

Metaflow

AWS EMR

AWS Batch

SQL

NoSQL

PostgreSQL

Docker

Kubernetes

Data quality

Tools

AWS EMR

AWS Batch

Spark

Metaflow

Job description

Rad AI is seeking a Senior Data Engineer in San Francisco to design and maintain scalable data pipelines, data architecture, and data products. You will collaborate with researchers to align data needs with model training and production monitoring.

You will own key projects, ensure data quality and security, and continuously improve the data infrastructure with modern tools and workflows.