Senior Data Engineer, Observability & Databricks Lakehouse

Koitecc Solutions; New York, Northern (NY, KY); Hybrid; USD 93,000 - 222,000; Full time

Benefits offered by this job

Medical, Dental, Vision

Paid Time Off

401(k) Matching

Job summary

CVS Health is seeking a Senior Data Engineer, Observability Engineering to design, build, and operate scalable Databricks-based data pipelines powering enterprise observability, security analytics, and operational intelligence. You will own PySpark workloads, governance standards, and data quality across telemetry, infrastructure, and applications, collaborating with Security and Platform teams.

The role emphasizes engineering excellence, reliability, and cost-efficient data platform operations

Qualifications

• 5+ years building enterprise data engineering solutions with Python, Spark, SQL.
• 5+ years designing and operating large-scale data pipelines across multiple sources.
• 4+ years working with Databricks in production environments, including Delta Lake and Unity Catalog.
• 3+ years developing high-volume batch and streaming data solutions with data quality controls.
• 2+ years collaborating with engineering teams and communicating solutions to stakeholders.

Responsibilities

• Design, build, deploy, and support scalable Databricks-based data pipelines and lakehouse solutions ingesting data from multiple sources.
• Develop and optimize PySpark workloads and Data lake architectures ensuring performance and reliability.
• Partner with Security, Infrastructure, Observability, and Platform Eng teams to onboard data sources and enforce governance.
• Contribute to engineering standards, code reviews, CI/CD practices, monitoring, and platform optimization.
• Promote best practices for Databricks, Delta Lake, Unity Catalog, and security controls.

Skills

Python

Spark

PySpark

SQL

Databricks

Delta Lake

Unity Catalog

Structured Streaming

Data pipelines

CI/CD

Education

Bachelor's degree

Tools

GitHub Actions

Azure DevOps

Databricks Asset Bundles

Job description

CVS Health is seeking a Senior Data Engineer, Observability Engineering to design, build, and operate scalable Databricks-based data pipelines powering enterprise observability, security analytics, and operational intelligence. You will own PySpark workloads, governance standards, and data quality across telemetry, infrastructure, and applications, collaborating with Security and Platform teams.

The role emphasizes engineering excellence, reliability, and cost-efficient data platform operations