Senior Data Engineer II: Cloud Data Platform & Mentorship

Compass; New York (NY); On-site; USD 168,000 - 180,000; Full time

Benefits offered by this job

Incentive programs

Paid vacation/holidays

Medical benefits

401(k)

FSAs

Commuter program

Life and disability insurance

Maven

Carrot

UrbanSitter

Employee Assistance Program

Pet insurance

Job summary

Compass is building a scalable analytics platform to empower data stakeholders across the company. As a Data Engineer, you will design fault-tolerant cloud data solutions, optimize data pipelines, and ensure cost-effective, secure data delivery.

You will mentor junior engineers and collaborate with analysts to align infra with business goals. You will work with Databricks, Spark, Kafka, Airflow, and AWS/GCP/Azure, applying IaC practices and enforcing governance standards.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, IT, or a related field.
• 8+ years of cloud data platform development experience (Databricks, Spark, Kafka, Airflow).
• 8+ years of programming experience in Python, Java, or C#.
• 3+ years using IaC tools like CloudFormation or Terraform.
• Expertise in data governance and automation; strong SQL and cloud platform skills.

Responsibilities

• Design and implement fault-tolerant, scalable cloud data solutions with cost-aware performance.
• Oversee platform health, updates, and feature adoption; drive operational efficiency.
• Implement data security practices and regulatory compliance (e.g., CCPA).
• Stay current with new data engineering technologies and opportunities for improvement.
• Mentor junior and mid-level engineers through code reviews and guidance.
• Collaborate with analysts, engineers, and stakeholders to align data infra with business goals.

Skills

Databricks

Spark

Kafka

Airflow

Python

Java

C#

SQL

Cloud platforms (AWS/GCP/Azure)

Infrastructure as Code

Governance & Compliance

Education

Bachelor’s/Master’s in CS/IT or related field

Job description

Compass is building a scalable analytics platform to empower data stakeholders across the company. As a Data Engineer, you will design fault-tolerant cloud data solutions, optimize data pipelines, and ensure cost-effective, secure data delivery.

You will mentor junior engineers and collaborate with analysts to align infra with business goals. You will work with Databricks, Spark, Kafka, Airflow, and AWS/GCP/Azure, applying IaC practices and enforcing governance standards.