Observability Data Engineer — Databricks Lakehouse Expert

CVS Health Corporation; New York (NY); Hybrid; USD 93,000 - 222,000; Full time

Benefits offered by this job

Medical insurance

Dental coverage

Vision coverage

Retirement plan

Job summary

CVS Health is seeking a Senior Data Engineer, Observability Engineering to design, build, and operate scalable Databricks-based data pipelines powering enterprise observability and security analytics. You will work with cross-functional teams to ingest, transform, govern, and deliver high-volume telemetry data that informs decision-making.

The role emphasizes engineering excellence, reliability, and cost-efficient data platform operations within the enterprise observability lakehouse framework.

Qualifications

• 5+ years of experience building and supporting enterprise data engineering solutions using Python, Spark, PySpark, SQL, and distributed data processing technologies.
• 5+ years of experience designing, developing, and operating large-scale data pipelines, data ingestion frameworks, and bronze/silver/gold lakehouse architectures across multiple data sources.
• 4+ years of experience working with Databricks in production environments, including Delta Lake, Unity Catalog, Databricks Asset Bundles, serverless SQL, performance optimization, and platform operations.
• 3+ years of experience developing high-volume batch and streaming data solutions, implementing data quality controls, troubleshooting pipeline issues, and improving reliability, scalability, and operational efficiency.
• 2+ years of experience collaborating with engineering teams, participating in code reviews, contributing to technical standards, applying cloud security and access management principles.

Responsibilities

• Design, build, deploy, and support scalable Databricks-based data pipelines and lakehouse solutions that ingest, transform, and curate observability, infrastructure, and security data from multiple enterprise sources.
• Develop and optimize PySpark workloads, Structured Streaming jobs, and bronze/silver/gold data architectures while ensuring high performance, reliability, scalability, data quality, and operational excellence.
• Partner with Security, Infrastructure, Observability, and Platform Engineering teams to onboard new data sources, establish governance standards, improve metadata management, and deliver trusted enterprise data products.
• Contribute to engineering standards, code reviews, CI/CD practices, monitoring strategies, and platform optimization efforts while proactively identifying and addressing technical debt, process gaps, and operational risks.
• Support the ongoing evolution of the enterprise observability lakehouse by promoting best practices for Databricks, Delta Lake, Unity Catalog, data governance, security controls, access management, and cost-efficient data platform operations.

Skills

Python

Spark

SQL

PySpark

Data pipelines

Education

Bachelor's degree

Tools

Databricks

Delta Lake

Unity Catalog

Job description

CVS Health is seeking a Senior Data Engineer, Observability Engineering to design, build, and operate scalable Databricks-based data pipelines powering enterprise observability and security analytics. You will work with cross-functional teams to ingest, transform, govern, and deliver high-volume telemetry data that informs decision-making.

The role emphasizes engineering excellence, reliability, and cost-efficient data platform operations within the enterprise observability lakehouse framework.