Senior Data Engineer, Databricks

Compass

United States

Hybrid

USD 140.000 - 200.000

Vollzeit

14 Tage+

Erhalte eine Antwort von diesem Arbeitgeber

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Benefits dieser Stelle

Meal allowance
Health insurance
Pet health insurance
Home office allowance
Welcome kit
Life insurance
Wellness support (Zenklub)
TotalPass
Continuous Learning program with top•s

Zusammenfassung

Compass is seeking a senior data engineer to design and evolve brand-new data architectures and scalable pipelines. You will optimize complex data workflows, implement batch and streaming processing, and work across Spark, Airflow, Kafka, and dbt in a cloud-centric environment.

Responsibilities include building Lakehouse models, leveraging Unity Catalog for governance, and delivering robust data ingestion and integrations using ADF and Lakeflow.

Qualifikationen

  • Experience defining and evolving data architectures using modern tools.
  • Proven ability to build scalable, high-performance data pipelines.
  • Strong knowledge of Databricks, Spark, and Delta Lake.
  • Hands-on with Azure Data Factory and data orchestration.
  • Familiarity with CI/CD, Git, and deployment automation.

Aufgaben

  • Define and evolve data architectures using Databricks, ensuring pipeline optimization and processing efficiency.
  • Build and maintain scalable data pipelines for high-quality data ingestion, transformation, and delivery.
  • Work with Spark, Airflow, Kafka, dbt, SQL/NoSQL databases, and cloud platforms (Databricks, AWS, BigQuery).
  • Design and implement Lakehouse models and Medallion Architecture (Bronze/Silver/Gold).
  • Implement batch and streaming data processing, including CDC and Structured Streaming.
  • Develop data ingestion/integration using ADF, Auto Loader, and Lakeflow.
  • Implement Unity Catalog governance, security, and access control.

Kenntnisse

Databricks
PySpark
Spark SQL
Delta Lake
Airflow
dbt
SQL/NoSQL
Python
REST APIs
Git & CI/CD
Unity Catalog
Azure Data Factory
Lakehouse
Delta table optimization

Tools

Databricks
Airflow
Kafka
dbt
PostgreSQL
MongoDB
Cassandra
AWS
BigQuery
Auto Loader
Lakeflow
Unity Catalog
Git

Jobbeschreibung

Job Responsibilities & Skills
  • Define and evolve data architectures using Databricks, ensuring pipeline optimization, processing efficiency, and resolution of highly complex technical issues.
  • Build and maintain scalable, reliable data pipelines using modern distributed processing technologies to ensure high-quality, high-performance data ingestion, transformation, and delivery.
  • Work with Apache Spark, Airflow, Kafka, dbt, SQL/NoSQL databases (PostgreSQL, MongoDB, Cassandra), and cloud platforms (Databricks, AWS Glue, BigQuery).
  • Design, define, and implement scalable data architectures using the Lakehouse model and Medallion Architecture (Bronze, Silver, and Gold).
  • Implement batch and streaming data processing solutions, including CDC and Structured Streaming.
  • Develop and orchestrate data ingestion and integration processes using Azure Data Factory (ADF), Auto Loader, and Lakeflow.
  • Implement data governance, security, and access control practices through Unity Catalog.
  • Develop REST APIs, integrations, and automations using Python.
  • Implement CI/CD, version control, and deployment automation processes using Git and Databricks Workflows.
  • Advanced proficiency in Databricks, PySpark, Spark SQL, and Delta Lake.
  • Strong experience with DLT / Lakeflow Spark Declarative Pipelines, CDC, and Structured Streaming.
  • Hands-on experience with Liquid Clustering, Auto Loader, and Delta table optimization.
  • Expertise in Unity Catalog, governance, security, lineage, and access control.
  • Experience defining and implementing data architecture, including the Lakehouse model and Medallion Architecture (Bronze, Silver, and Gold).
  • Hands-on experience with Azure Data Factory (ADF) for orchestration, data ingestion, and integration.
  • Experience developing REST APIs and advanced Python programming for building integrations, microservices, and automations.
  • Advanced knowledge of Spark performance tuning, Photon, partitioning, shuffle, and execution plan analysis.
  • Experience with Databricks Workflows, CI/CD, Git, and Declarative Automation Bundles.
  • Ability to optimize workloads, scale compute resources, and reduce processing costs.
  • Experience with observability, monitoring, and supporting production pipelines.
  • Ability to serve as a technical reference, mentor professionals, and establish data engineering best practices.
  • Certifications: Databricks Certified Data Engineer Associate; Databricks Certified Data Engineer Professional.
Benefits
  • Meal and/or food allowance
  • Health and dental insurance
  • Health insurance for your pet
  • Home office allowance (for remote positions)
  • Welcome kit as part of onboarding
  • Life insurance
  • Zenklub, offering psychological and nutritional support
  • TotalPass
  • Certification and Continuous Learning Program with partners (Adobe, Anthropic, AWS, Google, Microsoft, Salesforce)
  • Partnerships and discounts to support your professional development
  • Access to UOL content
  • Maternity kit
  • Extended maternity and paternity leave
  • Employee referral program
Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Senior Data Engineer Hot
Senior Data Engineer Hot

Binariks Inc. • USA

Hybrid
USD 120.000 - 180.000
Paid vacation
Public holidays
Sick leave
+10
Senior Forward Deployed Engineer - Central & Eastern NEW
Senior Forward Deployed Engineer - Central & Eastern NEW

The Briefed • Dallas (TX)

Vor Ort
USD 182.000 - 250.000
Healthcare & Wellbeing
401(k) plan
Flexible PTO
+1
Senior Data Engineer
Senior Data Engineer

Mom's Meals | A Purfoods Company • USA

Remote
USD 140.000 - 190.000
401(k) match
Life insurance
Short-term disability
+6
Senior Azure Data Engineer
Senior Azure Data Engineer

Yoh, A Day & Zimmermann Company • Plano (TX)

Vor Ort
USD 105.000 - 150.000
Medical, Prescription, Dental & Vision
Health Savings Account (HSA)
Life & Disability Insurance
+6
Databricks Architect
Databricks Architect

Tata Consultancy Services Limited • Marlborough (MA)

Vor Ort
USD 110.000 - 140.000
Discretionary Annual Incentive
Comprehensive Medical Coverage
401K Plan
+1
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

Capgemini • New York (NY)

Vor Ort
USD 89.000 - 166.000
Paid time off
Health benefits
401(k) plan
+1
Senior Forward Deployed Engineer - Field Tech Lead NEW
Senior Forward Deployed Engineer - Field Tech Lead NEW

The Briefed • New York (NY)

Vor Ort
USD 182.000 - 250.000
Healthcare coverage
401(k) + equity grants
Flexible time off
Databricks Subject Matter Expert
Databricks Subject Matter Expert

Jobgether • USA

Remote
USD 135.000 - 165.000
Health insurance
401(k) matching
Paid time off
+1
Databricks Subject Matter Expert
Databricks Subject Matter Expert

Jobgether • Oregon

Hybrid
USD 128.000 - 173.000
Remote work
Health insurance
Dental insurance
+8
Intermediate Data Engineer
Intermediate Data Engineer

abinbev • USA

Remote
USD 90.000 - 160.000
Performance and attendance bonuses
Pension
Meal and transport allowances
+7