Staff Data Engineer

Veritas Search Group; Carlsbad (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

An innovative firm is seeking a Staff Data Engineer to lead the design and implementation of scalable data systems that drive AI solutions. In this pivotal role, you will architect data pipelines, ensure compliance with regulations, and optimize performance in a secure environment. Collaborating with a talented team, you will have the opportunity to shape data strategy and make a real impact through mission-driven work. If you're passionate about leveraging data for meaningful change and thrive in a fast-paced environment, this is the perfect role for you.

Qualifications

• 8+ years of experience in data engineering with a focus on scalable infrastructure.
• Expertise in designing data pipelines and ensuring compliance with data privacy laws.

Responsibilities

• Architect and maintain secure data pipelines for batch and real-time processing.
• Mentor junior engineers and guide teams on best practices in data engineering.

Skills

Data Engineering

Python

SQL

NoSQL

HIPAA Compliance

Kafka

AWS

Data Governance

Tools

Snowflake

Redshift

BigQuery

Aurora

Docker

Kubernetes

Apache Spark

Hadoop

Flink

Job description

This role requires candidates who are currently authorized to work in the U.S. without sponsorship, and C2C arrangements are not accepted.

About the Role

Are you ready to build and scale the data infrastructure for a mission-driven organization using cutting-edge AI and data solutions? This Staff Data Engineer role is your chance to be at the forefront of designing and implementing high-performance, scalable data systems that power innovative AI-driven solutions.

Reporting directly to the Head of AI, you will play a foundational role in architecting data pipelines, optimizing performance, and ensuring regulatory compliance in a highly secure environment. This position is ideal for a hands-on technical leader who thrives in fast-paced, small-team environments and is passionate about leveraging data for real-world impact.

What You’ll Do

• Design Scalable Data Systems: Architect and maintain secure, efficient data pipelines for batch and real-time processing.
• Ensure Compliance: Implement HIPAA-compliant de-identification and data governance strategies for sensitive information.
• Enable AI/ML: Deliver clean, structured data for LLM prompting, model training, and analytics workflows.
• Enhance Data Logging: Build systems to capture and analyze key interactions for downstream processing.
• Optimize Performance: Design Snowflake schemas and optimize query performance for scalability and low latency.
• Provide Leadership: Mentor junior engineers and guide cross-functional teams on data engineering best practices.
• Monitor & Validate: Establish data quality frameworks, detect anomalies, and ensure system reliability.
• Drive Strategy: Define and execute a roadmap for scalable data infrastructure, supporting AI and recommendation systems.

What You Bring

Must-Have Skills & Experience

• 8+ years of professional experience in data engineering, with a proven track record in designing and scaling robust data infrastructure.
• Expertise in data pipeline architecture, including real-time streaming (Kafka, Kinesis) and batch workflows.
• Hands-on experience with Snowflake, Redshift, BigQuery, or Aurora, including schema design and query optimization.
• Strong programming skills in Python and SQL / NoSQL databases.
• Deep knowledge of HIPAA compliance, data privacy, and de-identification techniques.
• Experience working with distributed data frameworks (Apache Spark, Hadoop, Flink).
• Cloud expertise (primarily AWS, but familiarity with Azure is a plus).
• Experience with containerization (Docker) and orchestration tools (Kubernetes).

Nice-to-Haves

• Experience with AI/ML workflows, including feature stores and preprocessing pipelines.
• Prior experience working in a startup or fast-paced environment.
• Knowledge of graph databases and search & recommendation systems.
• Ability to set up best practices for data engineering from concept to full deployment.
• Strong communication skills—able to translate complex technical concepts into real-world business impact.

Why This Role?

• High Impact & Ownership – Play a pivotal role in shaping data strategy and AI initiatives from the ground up.
• Mission-Driven Work – Build AI-powered solutions that make a real difference in people's lives.
• Collaborative Culture – Work alongside a talented leadership team that values innovation, creativity, and teamwork.
• Career Growth – Be at the forefront of AI and emerging data technologies, expanding your expertise in data engineering and machine learning.