Senior Data Engineer: Spark Pipelines & Delta Lake

Visa Hunt; United States; On-site; USD 80,000 - 185,000; Full time

Job summary

Comcast Advertising is seeking a Data Engineer to build and optimize large-scale data pipelines using SQL, Python, and Spark on AWS. You will design robust data lake architectures with Delta Lake and Iceberg, empower self-service analytics, and drive data quality and governance across the platform.

The role emphasizes hands-on build, collaboration with Product and Engineering, and contributing to a modern, scalable data platform used by many internal teams and partners.

Qualifications

• Experience building large-scale data pipelines and data architectures.

Responsibilities

• Architect high-performance data pipelines for large-scale processing.
• Develop Big Data ETL workflows using modern frameworks like Delta Live Tables.
• Design data lake architectures with Delta Lake and Apache Iceberg.
• Define and enforce data engineering best practices for quality and performance.
• Lead observability across pipelines including cataloging and lineage.
• Produce thorough documentation of data flows and architectures.

Skills

SQL

Python

Apache Spark

AWS

Databricks

Delta Lake

Iceberg

Looker

Education

Bachelor's Degree

Tools

Databricks

Snowflake

Kubernetes

Teradata

Job description

Comcast Advertising is seeking a Data Engineer to build and optimize large-scale data pipelines using SQL, Python, and Spark on AWS. You will design robust data lake architectures with Delta Lake and Iceberg, empower self-service analytics, and drive data quality and governance across the platform.

The role emphasizes hands-on build, collaboration with Product and Engineering, and contributing to a modern, scalable data platform used by many internal teams and partners.