Senior Data Engineer, Observability Platform (Databricks)

Koitecc Solutions; Illinois; Hybrid; USD 93,000 - 222,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

Paid time off

Retirement savings options

Wellness programs

Job summary

CVS Health is seeking a Senior Data Engineer, Observability Engineering to advance enterprise observability, infrastructure, and security data capabilities. You will design, build, and operate scalable Databricks-based pipelines and lakehouse solutions ingesting telemetry, infrastructure, and security data.

The role emphasizes hands-on data engineering with ownership of reliability, performance, and best practices across data pipelines and governance frameworks.

Qualifications

• 5+ years of experience building and supporting enterprise data engineering solutions using Python, Spark, PySpark, SQL, and distributed data processing technologies.
• 5+ years of experience designing, developing, and operating large-scale data pipelines, data ingestion frameworks, and bronze/silver/gold lakehouse architectures across multiple data sources.
• 4+ years of experience working with Databricks in production environments, including Delta Lake, Unity Catalog, Databricks Asset Bundles, serverless SQL, performance optimization, and platform operations.
• 3+ years of experience developing high-volume batch and streaming data solutions, implementing data quality controls, troubleshooting pipeline issues, and improving reliability, scalability, and operational efficiency.
• 2+ years of experience collaborating with engineering teams, participating in code reviews, contributing to technical standards, applying cloud security and access management principles, and communicating technical solutions to both technical and non-technical stakeholders.

Responsibilities

• Design, build, deploy, and support scalable Databricks-based data pipelines and lakehouse solutions that ingest, transform, and curate observability, infrastructure, and security data from multiple enterprise sources.
• Develop and optimize PySpark workloads, Structured Streaming jobs, and bronze/silver/gold data architectures while ensuring high performance, reliability, scalability, data quality, and operational excellence.
• Partner with Security, Infrastructure, Observability, and Platform Engineering teams to onboard new data sources, establish governance standards, improve metadata management, and deliver trusted enterprise data products.
• Contribute to engineering standards, code reviews, CI/CD practices, monitoring strategies, and platform optimization efforts while proactively identifying and addressing technical debt, process gaps, and operational risks.
• Support the ongoing evolution of the enterprise observability lakehouse by promoting best practices for Databricks, Delta Lake, Unity Catalog, data governance, security controls, access management, and cost-efficient data platform operations.

Skills

Python

Spark

PySpark

SQL

Databricks

Delta Lake

Unity Catalog

Performance optimization

CI/CD

GitHub Actions

Azure DevOps

Data pipelines

Security analytics

Education

Bachelor's degree

Tools

Databricks

Delta Lake

Unity Catalog

GitHub Actions

Azure DevOps

Job description

CVS Health is seeking a Senior Data Engineer, Observability Engineering to advance enterprise observability, infrastructure, and security data capabilities. You will design, build, and operate scalable Databricks-based pipelines and lakehouse solutions ingesting telemetry, infrastructure, and security data.

The role emphasizes hands-on data engineering with ownership of reliability, performance, and best practices across data pipelines and governance frameworks.