Senior Software Engineer, Big Data

ziprecruiter; United States; Remote; USD 170,000 - 240,000; Full time

Benefits offered by this job

Remote work

Hybrid option

401(k) plan

Job summary

ZipRecruiter is seeking a senior data engineer to build large-scale data systems powering a high-traffic job marketplace. The role blends data engineering, ML infrastructure, and distributed systems to serve millions of users and thousands of customers.

This is a fully remote US-based position ideal for engineers who enjoy evolving data-driven platforms and scalable architectures in a fast-moving environment.

Qualifications

• 5+ years of professional software development with Python and big data.
• Hands-on experience with Spark, Kafka, Hadoop or Hive.
• Strong CS fundamentals: OOP, data structures, algorithms.
• Experience with Docker and Kubernetes.
• Proven ability to write evolvable, instrumented distributed code.

Responsibilities

• Design and build data pipelines for ML and BI workloads.
• Develop and maintain ML infrastructure at scale.
• Deploy and operate cloud-based services for high-volume projects.
• Write, test, and deploy production code in Kubernetes.
• Contribute to architecture evolution of a growing distributed system.
• Collaborate on streaming and batch data systems processing petabytes.

Skills

Python

Big data

Distributed systems

Education

BS/MS/PhD in CS/Math/Physics or related field

Tools

Docker

Kubernetes

Spark

Kafka

Hadoop

Hive

Job description

Role overview

A senior engineering position focused on building large-scale data systems that power matching intelligence for a high-traffic employment marketplace. The role combines data engineering, ML infrastructure, and distributed systems work to serve millions of users and tens of thousands of business customers. It is a fully remote, US-based opportunity suited to engineers who enjoy evolving complex, data-driven platforms.

Responsibilities

• Design and build data processing and exploration pipelines that support machine learning and business intelligence workloads.
• Develop and maintain the ML infrastructure used to power marketplace intelligence at scale.
• Deploy and operate cloud-based services supporting critical, high-volume projects.
• Write, test, instrument, and deploy production code within a Kubernetes environment.
• Contribute to the ongoing innovation and architectural evolution of a growing distributed system.
• Collaborate on both streaming and batch data systems that process petabyte-scale datasets.

Requirements

• Five or more years of professional software development experience with a focus on Python and big data technologies.
• Hands-on experience with at least one of Spark, Kafka, Hadoop, or Hive, plus broader familiarity with the big data ecosystem.
• Strong computer science fundamentals, including object-oriented programming, data structures, and algorithms.
• Practical experience with containerization tools such as Docker and Kubernetes.
• Track record of writing evolvable, well-instrumented, and efficient code in distributed production systems.

Nice to have

• Eight or more years of professional software development experience centered on Python and big data.
• BS, MS, or PhD in Computer Science, Mathematics, Physics, or a closely related technical field, or equivalent practical experience.
• Experience with data integration tools such as Apache Flume and NiFi, or related streaming ingestion systems.
• Familiarity with data storage technologies including Delta Lake, HBase, Cassandra, and MongoDB.
• Exposure to modern data processing frameworks such as Apache Hudi, Apache Beam, Apache Flink, Google Cloud Dataflow, Amazon Kinesis Data Analytics, or Azure Databricks.

Benefits and work setup

• Competitive base compensation with a US salary range of $170,000 to $240,000, plus potential equity, bonus, or commission depending on the offer.
• Comprehensive medical, financial, and retirement benefits, including an employer-matched 401(k) plan.
• Flexible vacation and paid time off policy.
• Fully remote work available for most US-based candidates, with a hybrid option also offered.