Senior Platform Engineer: Spark Data Pipelines

InvestedintheMission; Washington (District of Columbia); Hybrid; USD 175,000 - 195,000; Full time

Benefits offered by this job

Competitive salary

Company Equity

Benefits package

Flexible Vacation

401(k) plan

Job summary

Quantifind is seeking a Senior Software Engineer to work on the Apache Spark pipeline powering risk signal results from diverse data sources. You will help define and deliver data services and ML infrastructure as part of the Platform Team in Washington, D.C.

You will work with Data Scientists, DevOps, Front-end Engineers and Product Managers, building scalable microservices, robust tests, and efficient data processing on distributed systems.

Qualifications

• BS or MS in Computer Science or equivalent.
• 5+ years as a professional Software Engineer with a focus on platform engineering.
• Professional experience at a software startup strongly preferred.
• Great communication skills to work with Data Scientists, DevOps, and Product Managers.
• Solid understanding of concurrency and distributed systems.
• Professional experience building software with Scala and/or Java.
• Experience working with code that runs in a JVM.
• Ability to design and prototype scalable algorithms.
• Experience with Spark, HDFS, Iceberg, Airflow, and/or MapReduce.
• Experience writing unit tests and automated integration tests.
• Familiarity with storage and caching solutions (Memcached, Redis, Postgres).
• Ability to build microservices with REST APIs for real-time query engines and web crawlers.
• Knowledge of deploying microservices in Kubernetes.
• Knowledge to set up ETL pipelines for data streams.
• Familiarity with Machine Learning techniques.
• Must be legally authorized to work in the United States without sponsorship.

Responsibilities

• Define and deliver data services and ML infrastructure as part of the Platform Team.
• Own backend powering SaaS products and process terabytes daily on Spark clusters.
• Collaborate with cross-functional teams to improve design, performance and testing of software.

Skills

Scala/Java

Distributed systems

Concurrency

REST APIs

Kubernetes

Spark

Unit tests

Education

BS or MS in Computer Science

Tools

Apache Spark

HDFS

Iceberg

Airflow

MapReduce

Job description

Quantifind is seeking a Senior Software Engineer to work on the Apache Spark pipeline powering risk signal results from diverse data sources. You will help define and deliver data services and ML infrastructure as part of the Platform Team in Washington, D.C.

You will work with Data Scientists, DevOps, Front-end Engineers and Product Managers, building scalable microservices, robust tests, and efficient data processing on distributed systems.