Lead Data Engineer - Remote, Databricks & AWS

Comcast Advertising; Reston, Northern (VA, KY); Hybrid; USD 158,000 - 237,000; Full time

Job summary

Comcast Advertising is seeking a Lead Data Engineer to design and manage advanced data engineering solutions that inform decision-making and boost operational efficiency. You will oversee data pipelines, storage systems, and data quality across ingestion, processing, and delivery to analytics platforms.

Collaborate with cross-functional teams to refine data sourcing, implement standardized APIs and views, and deploy solutions on Kubernetes, AWS, and Databricks to meet business needs.

Qualifications

• Bachelor's Degree preferred; combination of coursework and experience may be considered
• 10+ years of relevant work experience
• Strong data engineering and pipeline development experience

Responsibilities

• Leading data structure and pipeline development to standardize, transform, and deliver data for insights and reporting, following best practices
• Ensuring data quality throughout the lifecycle, from ingestion to processing and loading, through rigorous data quality frameworks
• Designing and implementing standardized methods for data consumption, including APIs, database views, and data extracts, to facilitate end-user access
• Designing and deploying data solutions on Kubernetes, Teradata, AWS, and Databricks to meet specific business requirements
• Choosing data storage solutions tailored to sensitivity and access needs using platforms like minIO, Teradata, AWS S3, and Redshift
• Mapping data lineage and applying transformation rules to streamline troubleshooting and impact analysis for data modifications
• Collaborating with cross-functional teams to refine data sourcing and processing, enhancing data quality, and optimizing operational processes
• Setting design standards and QA protocols for software/systems development to ensure data flows and storage solutions interoperate effectively
• Consistent exercise of independent judgment and discretion in matters of significance
• Regular attendance; ability to work nights and weekends as needed

Skills

Apache Spark

Databricks Platform

Data Engineering

Java

SQL

Education

Bachelor's Degree

Tools

Kubernetes

Teradata

AWS

Databricks

Redshift

Job description

Comcast Advertising is seeking a Lead Data Engineer to design and manage advanced data engineering solutions that inform decision-making and boost operational efficiency. You will oversee data pipelines, storage systems, and data quality across ingestion, processing, and delivery to analytics platforms.

Collaborate with cross-functional teams to refine data sourcing, implement standardized APIs and views, and deploy solutions on Kubernetes, AWS, and Databricks to meet business needs.