Senior Platform Engineer - Kubernetes & Spark

Comcast; West Chester (Chester County); On-site; USD 140,000 - 180,000; Full time

Job summary

Comcast in the United States is seeking a Sr. Platform Engineer to build, manage, and optimize the infrastructure that runs large-scale data workloads.

You will design systems for metrics collection with Prometheus and visualization with Grafana, ensuring a robust data processing environment for engineers and scientists. Responsibilities include managing Spark platforms on Kubernetes and AWS, packaging workloads with Docker, deploying with Terraform/Ansible, and writing Python/Scala.

Qualifications

• Bachelor's degree or equivalent experience in computer science or related field; typically 7 years in a DevOps or Systems Engineering role.
• Expertise in Apache Spark: deep understanding of Spark architecture, RDDs, DataFrames, execution hierarchy, lazy evaluation, shuffling, fault tolerance.
• Proficiency in Python, PySpark, and Scala/Java for Spark development and automation.
• Proficiency in Linux scripting (Bash).
• Proficient in SQL writing.
• Experience with CI/CD tools and GitHub.
• Experience with observability tools like Prometheus, Grafana.
• Strong knowledge of networking protocols (TCP/IP, DNS) and hardware basics.
• Automation via Terraform/Ansible; cloud/on-prem experience (AWS/Azure/GCP); Docker/Kubernetes.

Responsibilities

• Architect and manage Spark platforms on Kubernetes and cloud services (AWS EKS).
• Package Spark workloads and integrate with orchestration systems like Flyte.
• Deploy infrastructure using Terraform/Ansible.
• Troubleshoot and optimize job failures and resource issues for cost efficiency.
• Develop tools in Python/Java/Scala and shell scripting to boost productivity.
• Write medium to complex SQL queries as needed.
• Implement and maintain monitoring, logging, and alerting with Prometheus and Grafana.
• Develop data catalog capabilities (Iceberg, Unity Catalog) for authorization and lineage.
• Collaborate with Data Stewards, Analysts, and Scientists to address data needs.

Skills

Apache Spark

Python

Scala/Java

Linux Bash

SQL

CI/CD

Prometheus Grafana

Terraform/Ansible

AWS/Azure/GCP

Docker Kubernetes

Data Governance

Delta Lake

Apache Iceberg

Apache Kafka

Databricks

Unity Catalog

Alluxio

IAM/VPC

Networking TCP/IP

Hadoop

Education

Bachelor's degree in Computer Science or related field

Tools

Docker

Kubernetes

Databricks

Snowflake

Unity Catalog

Job description

Comcast in the United States is seeking a Sr. Platform Engineer to build, manage, and optimize the infrastructure that runs large-scale data workloads.

You will design systems for metrics collection with Prometheus and visualization with Grafana, ensuring a robust data processing environment for engineers and scientists. Responsibilities include managing Spark platforms on Kubernetes and AWS, packaging workloads with Docker, deploying with Terraform/Ansible, and writing Python/Scala.