Senior Data Engineer - Databricks Lakehouse Lead

REALIGN LLC; Irvine (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

REALIGN LLC is seeking a hands-on Senior Data Engineer to design, build, and optimize enterprise-scale data solutions on the Databricks Lakehouse Platform. You will develop reliable batch and streaming pipelines and modernize legacy ETL workloads for analytics, reporting, risk, regulatory, and investment-management use cases.

The ideal candidate brings deep Databricks, Spark, PySpark, SQL, Python, dbt, Airflow, Delta Lake, Unity Catalog, cloud storage, data quality, CI/CD, and production support

Qualifications

• Typically 7–10 years of data engineering, data warehousing, or distributed data‑processing experience.
• Strong hands‑on experience with Databricks, Apache Spark, PySpark, Delta Lake, Python, and advanced SQL.
• Experience building production‑grade ETL and ELT pipelines for large datasets.
• Experience with dbt Core or dbt Cloud, including models, macros, tests, documentation, and incremental processing.
• Experience with Apache Airflow, Astronomer, Databricks Workflows, or comparable orchestration platforms.
• Experience with Unity Catalog, data lineage, role‑based access, and data‑governance controls.
• Experience with cloud data services on AWS, Azure, or Google Cloud.
• Working knowledge of Git, CI/CD, automated testing, monitoring, and production‑support practices.
• Strong troubleshooting, communication, collaboration, and technical‑documentation skills.

Responsibilities

• Design, build, test, deploy, and maintain scalable ETL and ELT pipelines using Databricks, PySpark, Spark SQL, Python, and SQL.
• Develop reusable ingestion and transformation frameworks for structured, semi-structured, and streaming data.
• Implement batch, incremental, change-data-capture, and streaming processing patterns.
• Build and maintain Delta Lake tables using medallion architecture across Bronze, Silver, and Gold layers.
• Develop dbt models, tests, macros, packages, documentation, and incremental processing patterns.
• Create and maintain Apache Airflow DAGs and Databricks Workflows with dependency management, retries, alerting, and operational controls.
• Collaborate to integrate data from APIs, databases, files, event streams, and cloud services.
• Provide technical designs, mapping specs, lineage docs, deployment instructions, and runbooks.

Skills

Databricks

Apache Spark

PySpark

SQL

Python

dbt

Apache Airflow

Delta Lake

Unity Catalog

Cloud storage

Data quality

CI/CD

Production support

Tools

Databricks

dbt Core

dbt Cloud

Apache Airflow

Astronomer

Databricks Workflows

Terraform

Job description

REALIGN LLC is seeking a hands-on Senior Data Engineer to design, build, and optimize enterprise-scale data solutions on the Databricks Lakehouse Platform. You will develop reliable batch and streaming pipelines and modernize legacy ETL workloads for analytics, reporting, risk, regulatory, and investment-management use cases.

The ideal candidate brings deep Databricks, Spark, PySpark, SQL, Python, dbt, Airflow, Delta Lake, Unity Catalog, cloud storage, data quality, CI/CD, and production support