Staff Data Engineer

ZipRecruiter; Carlsbad (CA); On-site; USD 140,000 - 160,000; Full time

Job summary

An innovative firm is seeking a Staff Data Engineer to spearhead the development of high-performance data pipelines and scalable APIs. In this pivotal role, you will leverage your expertise in Databricks and Rust to design and optimize data infrastructure, ensuring efficient data processing and real-time analytics. You will mentor engineers, drive architectural decisions, and collaborate with diverse teams to implement best practices in data governance and security. This position offers an exciting opportunity to shape data solutions in a dynamic environment where your contributions will significantly impact the organization's success.

Qualifications

• 5-7 years of experience in data engineering with Azure expertise.
• Strong experience with Databricks and Apache Spark for data solutions.
• Proficiency in Rust for backend development and API design.

Responsibilities

• Design and maintain scalable data pipelines on Databricks.
• Lead the development of data infrastructure and optimize performance.
• Mentor junior engineers and collaborate with cross-functional teams.

Skills

Data Engineering

Azure Data Services

Databricks

Apache Spark

Rust

Python

SQL

Communication Skills

Microservices Architecture

Machine Learning Data Pipelines

Education

Bachelor's Degree in Computer Science

Master's Degree in STEM

Tools

Databricks

Apache Spark

Docker

Kubernetes

Kafka

Pulsar

Job description

Job Description

We are looking for an experienced Staff Data Engineer to lead the development of our high-performance data pipelines and scalable APIs using Databricks and Rust. As a key technical leader, you will design, implement, and optimize data infrastructure, ensuring efficient data processing and real-time analytics capabilities. This role requires deep expertise in big data frameworks, distributed systems, and backend engineering, as well as a strong ability to mentor engineers and drive architectural decisions.

Essential Duties and Responsibilities

1. Design and Develop Data Pipelines: Design, build, and maintain scalable, high-performance data pipelines and ETL processes on Databricks/Microsoft Azure to acquire, ingest, process, and transform data from various sources into structured, usable formats.
2. Azure Data Platform Expertise: Leverage your knowledge of Microsoft Azure services such as Azure Databricks, Azure Storage, and Azure SQL Database to architect and optimize data solutions.
3. Architect and develop large-scale data pipelines: Use Databricks and Apache Spark (PySpark/Scala) for batch and streaming workloads.
4. Design, implement, and optimize Rust APIs: Support data access, ingestion, and real-time processing.
5. Lead data infrastructure development: Ensure reliability, security, and cost efficiency.
6. Integrate data sources: Connect Databricks with cloud data lakes (Azure Data Lake, AWS S3, GCP BigQuery) and databases (MongoDB, Postgres).
7. Data Governance and Security: Implement best practices for data governance, security, compliance, and privacy.
8. Optimize Spark jobs: Improve efficiency and cost-effectiveness.
9. Standards and Compliance: Define and enforce data governance, quality, security standards, including access control and encryption.
10. Develop real-time pipelines: Use Kafka, Pulsar, or similar messaging systems for event-driven data processing.
11. Promote best practices: Ensure robust API security, logging, and monitoring for Rust-based backend services.
12. Mentor engineers: Guide junior and mid-level engineers on system design, performance tuning, and architecture.
13. Collaborate cross-functionally: Work with data scientists, software engineers, and DevOps teams.
14. Documentation and training: Create technical documentation, best practices, and training materials.
15. Research and innovation: Stay updated on industry trends and identify opportunities for technological improvements.

Requirements

The following are the minimum requirements for this role:

• 5-7 years of experience in data engineering or related fields, with expertise in Azure data services.
• Strong experience with Databricks and Apache Spark (PySpark or Scala).
• Proficiency in Rust for backend/API development.
• Experience designing and deploying data solutions on Microsoft Azure.
• Knowledge of microservices architecture (Docker, Kubernetes).
• Proficiency in Python, SQL, Scala, Rust.
• Excellent communication skills, capable of explaining technical concepts clearly to diverse audiences.
• Experience with Delta Lake, Apache Iceberg, or similar data lakes.
• Knowledge of GraphQL and gRPC for data APIs.
• Experience with machine learning data pipelines and feature stores.
• Ability to analyze complex data requirements and troubleshoot issues.
• Experience with medical imaging data (X-ray, CT, MRI) and DICOM format is a plus.
• Certifications such as Azure Data Engineer Associate or Azure Solutions Architect are highly desirable.
• Proven leadership and mentoring experience.
• Strong analytical skills.

Education and Experience

• Bachelor’s degree in computer science or a STEM field; a master’s degree is a plus.

Note: For U.S. roles requiring hospital access, candidates must meet credentialing and vaccination requirements.

ATEC is committed to equal employment opportunity and will provide reasonable accommodations as required by law.

Salary Range

Full-Time Annual Salary: $140,000 to $160,000, depending on qualifications, experience, and internal equity.