Senior Software Engineer, Data Ingestion

Momentum

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Slack is seeking a Senior Software Engineer for the Data Ingestion Team within the Data Engineering org. You will design and operate the platform that ingests data from diverse sources into our Lakehouse using streaming and batch approaches, handling 180PB+ storage and 1T records/day across 400+ pipelines and 1500+ Airflow DAGs.

You’ll lead initiatives to scale streaming and lakehouse strategies, build scalable services, mentor engineers, and collaborate with cross-functional partners to deliver

Qualifications

  • 7+ years of software/data engineering experience in streaming and lakehouse tech.
  • Experience with building apps using Kafka, Flink or Spark Streaming, Iceberg/Hudi/Delta.
  • Hands-on with Spark, Airflow, EMR, and AWS S3 for batch pipelines.
  • Proficient in SQL, Python, Java/Scala, Go.
  • Experience crafting distributed microservices with Kubernetes, Docker, AWS ECS/EKS, Terraform, and observability tools.

Responsibilities

  • Design and develop highly scalable streaming and batch pipelines for ingesting and processing datasets.
  • Design and implement Slack’s Lakehouse solutions and enable production use cases for customers.
  • Improve the reliability and performance of services/pipelines with AI-assisted/Agentic solutions.
  • Automate dataset lifecycle: schema evolution, metadata management, backfill, deprecation, migration.
  • Lead technical initiatives end to end and collaborate with cross-functional partners.
  • Mentor and coach engineers across the team and org.

Skills

Kafka
Spark Streaming
Airflow
S3
Kubernetes
Docker
Terraform
Iceberg
Delta

Tools

Kafka Connect
Flink
Spark

Job description

The Experience

Slack is looking for a senior software engineer to join the Data Ingestion Team. As part of the Data Engineering organization, we build and operate the platform that ingests data from various sources into our Lakehouse with both streaming and batch approaches. Currently, our infrastructure handles 180PB+ of storage, processes ~1T records a day, 400+ data pipelines and 1500+ Active Airflow DAGs. As Slack’s data grows (along with the number of customers, features and employees), the goal of the team is to build a highly scalable and resilient ingestion platform leveraging modern Streaming and Lakehouse technologies.

As a Senior Engineer, you will play a key role in driving our streaming and lakehouse strategies while also building highly scalable services and tools to help our customers acquire large volumes of high quality data at scale and making customers’ interactions with the ingested data pleasant and productive. You will have deep technical skills, be a self-starter, detail and quality oriented, and passionate about driving data driven decisions and having a huge impact at Slack!

What You'll Actually be Doing:
  • Design and develop highly scalable and resilient streaming and batch pipelines for ingesting and processing structured and unstructured datasets
  • Design and implement Slack’s Lakehouse solutions and enable production use cases for customers
  • Improve the reliability and performance of the services/pipelines through AI assisted and/or Agentic solutions and tools.
  • Automate and handle the lifecycle of datasets (schema evolution, metadata management, change and backfill management, deprecation and migration).
  • As one of the tech leads, comfortably collaborate with cross functional partners and lead technical initiatives independently end to end.
  • Be a role model and a multiplier, coaching and mentoring other engineers across the team and the org.
  • Write, review, or provide feedback on a technical design proposal from others.
You're our person if you have:
  • U.S. citizenship and willing to undergo a background check for GovSlack authorization. We can not provide visa sponsorship for this role.
  • 7+ years of software/data engineering experience, including experience with building applications using Streaming and Lakehouse technologies, e.g. Kafka/Kafka Connect, Flink/Spark Streaming, Iceberg/Hudi/Delta or equivalent
  • You have hands on experience and knowledge on building and maintaining batch data pipelines using Spark, Airflow, EMR, S3 or equivalent
  • You are proficient in object-oriented and/or functional programming languages: SQL, Python, Java/Scala, Go or equivalent
  • You are skilled at crafting and building robust distributed microservices with tools like Kubernetes, Docker, AWS ECS/EKS, Terraform, Grafana, etc.
Even Better If:
  • You are familiar with AI-assisted software development and automation and have hands on experience of Claude Code/Codex or equivalent
  • You have excellent written and verbal communication and interpersonal skills; able to effectively collaborate with cross functional partners and explaining sophisticated technical concepts to non-technical stakeholders
  • You have high growth expectations for yourself and your team, and a willingness to push yourself and your team to achieve them
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior/Staff Software Engineer, Data Infrastructure
Senior/Staff Software Engineer, Data Infrastructure

Momentum • San Francisco (CA)

On-site
USD 180,000 - 260,000
Senior Data Ingestion Engineer - Streaming & Lakehouse
Senior Data Ingestion Engineer - Streaming & Lakehouse

Momentum • San Francisco (CA)

On-site
USD 180,000 - 240,000
Senior Software Engineer, Data Ingestion
Senior Software Engineer, Data Ingestion

Slack • Atlanta (GA)

On-site
USD 173,000 - 260,000
Senior Data Ingestion Engineer - Lakehouse & Streaming
Senior Data Ingestion Engineer - Lakehouse & Streaming

Slack • San Francisco (CA)

On-site
USD 173,000 - 286,000
Senior Software Engineer, Data Ingestion
Senior Software Engineer, Data Ingestion

Slack • Seattle (WA)

On-site
USD 173,000 - 260,000
Senior Software Engineer, Data Ingestion
Senior Software Engineer, Data Ingestion

Slack • Bellevue (WA)

On-site
USD 173,000 - 260,000
Staff Data Engineer - Slack
Staff Data Engineer - Slack

Momentum • San Francisco (CA)

On-site
USD 180,000 - 240,000
Senior Director, Software Engineering - Slack Application Architecture and Systems
Senior Director, Software Engineering - Slack Application Architecture and Systems

Momentum • San Francisco (CA)

On-site
USD 260,000 - 400,000
Senior Data Ingestion Engineer - Streaming & Lakehouse
Senior Data Ingestion Engineer - Streaming & Lakehouse

Slack • Seattle (WA)

On-site
USD 173,000 - 260,000
Senior Data Ingestion Engineer – Streaming & Lakehouse
Senior Data Ingestion Engineer – Streaming & Lakehouse

Slack • Bellevue (WA)

On-site
USD 173,000 - 260,000