Data Engineer: Large-Scale Spark & Streaming Pipelines

Socket.dev; Austin (TX); On-site; USD 130,000 - 190,000; Full time

Job summary

Apple Ads data engineering team seeks a Software Engineer to build and operate production data pipelines using Spark, Kafka, and cloud storage. You will design distributed pipelines for batch, near-real-time, and streaming workloads with attention to correctness, privacy, reliability, and performance.

The ideal candidate has hands-on Spark experience, strong CS fundamentals, and a track record of delivering scalable data infrastructure.

Qualifications

• 3+ years in professional software or data engineering on production systems.
• Strong CS fundamentals: data structures, algorithms, concurrency, distributed systems.
• Production-quality Java and/or Scala software development.
• Hands-on with large-scale data pipelines using Apache Spark.
• Deep Spark knowledge: partitioning, shuffles, joins, caching, memory, performance.
• Experience designing distributed batch/streaming systems with Kafka/Hadoop/S3.
• Strong SQL skills on large analytical datasets.
• Knowledge of data modeling, partitioning, schema evolution, Parquet formats.
• Experience building reliable production systems with testing, monitoring, and ops.

Responsibilities

• Design, develop, and operate production data pipelines at scale.
• Own end-to-end data solutions from architecture to deployment and observability.
• Collaborate with engineers, data scientists, SREs, and product teams to translate requirements.

Skills

Java

Scala

Distributed systems

Spark

Kafka

SQL

Data modeling

Troubleshooting

Education

Bachelor's degree in CS/CE/SE or equivalent

Tools

Apache Spark

Kafka

Hadoop

S3/ object storage

Iceberg

Kubernetes

Job description

Apple Ads data engineering team seeks a Software Engineer to build and operate production data pipelines using Spark, Kafka, and cloud storage. You will design distributed pipelines for batch, near-real-time, and streaming workloads with attention to correctness, privacy, reliability, and performance.

The ideal candidate has hands-on Spark experience, strong CS fundamentals, and a track record of delivering scalable data infrastructure.