Senior Data Engineer — Databricks Lakehouse

Mcgraw Hill; Columbus (OH); On-site; USD 130,000 - 170,000; Full time

Job summary

McGraw Hill is seeking a Senior Engineer - Data Engineering (Databricks) to own data pipelines and analytics models powering reports for Open Learning products. You will work with Databricks lakehouse, Unity Catalog governance, and a cross-functional team across product and data platform domains.

Responsibilities include building scalable PySpark data pipelines, designing dimensional models, and optimizing the SQL serving layer while ensuring data quality and cost efficiency in a fast-paced

Qualifications

• 5+ years of software or data engineering experience, including 3+ years building production data pipelines on Databricks or an equivalent Spark platform in AWS
• Experience delivering production-grade, fault-tolerant software in a fast-paced environment with a distributed team
• Advanced SQL skills, including tuning complex analytical queries over large datasets
• Expertise in data warehousing, lakehouse concepts, data modeling, and cloud data architecture
• Strong applied knowledge of several of: Unity Catalog governance, batch and incremental ETL/ELT, Spark performance tuning, REST services, AWS data services (Athena, Glue, EMR, Lambda), Terraform, GitHub Actions, and Git
• A Bachelor's degree in Computer Science or a related field, or equivalent experience

Responsibilities

• Translate business requirements into technical specifications and designs, partnering with product and engineering leaders
• Build and maintain scalable batch and incremental data pipelines using PySpark, Spark SQL, Delta Lake, and Databricks Workflows
• Design reporting data models using dimensional modeling and medallion (bronze/silver/gold) architecture
• Build and optimize the SQL serving layer in Unity Catalog, including views, materialized views, and table-valued functions
• Tune Spark and SQL performance and manage compute cost
• Own data quality, observability, and reliability through automation, testing, and monitoring
• Apply data governance and access controls through Unity Catalog
• Lead design reviews and mentor engineers on data engineering best practices

Skills

PySpark

Spark SQL

Delta Lake

Databricks

SQL

Data modeling

Unity Catalog

ETL/ELT

AWS

GitHub Actions

Education

Bachelor's degree in Computer Science or related field

Tools

GitHub Actions

Docker

Kubernetes

Terraform

Databricks

Job description

McGraw Hill is seeking a Senior Engineer - Data Engineering (Databricks) to own data pipelines and analytics models powering reports for Open Learning products. You will work with Databricks lakehouse, Unity Catalog governance, and a cross-functional team across product and data platform domains.

Responsibilities include building scalable PySpark data pipelines, designing dimensional models, and optimizing the SQL serving layer while ensuring data quality and cost efficiency in a fast-paced