Senior Data Engineer — AI-Driven Pipelines & Cloud

Vatica Health; United States; Remote; USD 140,000 - 180,000; Full time

Benefits offered by this job

Competitive salary

401k plans

Growth opportunities

Generous PTO

Excellent health plans

Job summary

Vatica Health is seeking a Senior Data Engineer to design, build, and operate data pipelines and APIs powering our data-driven products. You will own data movement from ingestion to consumption, ensuring reliability and security, and collaborate with product, analytics, data science, and ML teams.

The role involves leading technical reviews, establishing engineering standards, and mentoring other engineers while building scalable AWS-based cloud solutions and AI-enabled workflows for risk

Qualifications

• 8–10 years of professional experience in data engineering or a related field.
• Strong Python and advanced SQL skills, with experience delivering production data pipelines at scale.
• Hands-on experience with at least one workflow orchestration platform and big data processing frameworks.
• Experience developing and deploying production REST APIs, including integration, security, and automated testing.
• Experience with cloud platforms and data services.
• Understanding of data warehousing, lakehouse architecture, and data modeling.
• Working knowledge of AI and machine learning workflows, LLM APIs, embeddings, and vector databases.
• Experience with Git, CI/CD, Docker, and infrastructure as code tools such as Terraform.
• Strong problem-solving, communication, and stakeholder management skills.

Responsibilities

• Design, build, and maintain batch and streaming pipelines for structured and unstructured data from databases, APIs, event streams, and files.
• Develop ETL/ELT processes and optimize performance, cost, scalability, and reliability.
• Build and operate cloud solutions using AWS services such as EC2, ECS, Lambda, API Gateway, and SQS.
• Develop and manage data workflows, including scheduling, dependencies, retries, alerts, and backfills.
• Establish monitoring, logging, and observability to identify issues and support reliable production operations.
• Design, develop, test, and maintain secure REST APIs and Python-based data services using frameworks such as FastAPI or Flask.
• Implement API authentication, authorization, validation, versioning, error handling, rate limiting, and documentation.
• Build data foundations for machine learning and generative AI, including feature pipelines, training and inference datasets, embeddings, and retrieval-based solutions.
• Integrate AI tools and large language model (LLM) APIs into workflows for document processing, enrichment, classification, and summarization.
• Implement data quality checks, validation, lineage, and documentation, applying security, privacy, and compliance standards.
• Lead architecture and code reviews, mentor engineers, and communicate technical recommendations to business and technical stakeholders.

Skills

Python

SQL

Data pipelines

REST APIs

Cloud (AWS)

Data warehousing

AI/ML workflows

Git / CI/CD

Terraform

Docker

Education

Bachelor's or Master's in CS/Engineering

Tools

Terraform

Docker

Git

CI/CD tools

FastAPI/Flask

Job description

Vatica Health is seeking a Senior Data Engineer to design, build, and operate data pipelines and APIs powering our data-driven products. You will own data movement from ingestion to consumption, ensuring reliability and security, and collaborate with product, analytics, data science, and ML teams.

The role involves leading technical reviews, establishing engineering standards, and mentoring other engineers while building scalable AWS-based cloud solutions and AI-enabled workflows for risk