Senior Data Engineer - Databricks Lakehouse Expert

Acestack; Irvine (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Acestack in Irvine, CA is seeking a hands-on Senior Data Engineer to design, build, and optimize enterprise-scale data solutions on the Databricks Lakehouse Platform. You will develop batch and streaming pipelines, modernize ETL workloads, and deliver trusted data products for analytics and risk management.

The ideal candidate has deep experience with Databricks, Spark, PySpark, SQL, Python, dbt, Airflow, Delta Lake, and Unity Catalog, including production support and CI/CD. This role is onsite.

Qualifications

• Typically 7–10 years of data engineering, data warehousing, or distributed data-processing experience.
• Strong hands-on experience with Databricks, Apache Spark, PySpark, Delta Lake, Python, and advanced SQL.
• Experience building production-grade ETL and ELT pipelines for large datasets.
• Experience with dbt Core or dbt Cloud, including models, macros, tests, documentation, and incremental processing.
• Experience with Apache Airflow, Databricks Workflows, or comparable orchestration platforms.
• Experience with Unity Catalog, data lineage, role-based access, and data-governance controls.
• Experience with cloud data services on AWS, Azure, or Google Cloud.
• Working knowledge of Git, CI/CD, automated testing, monitoring, and production-support practices.

Responsibilities

• Design, build, test, deploy, and maintain scalable ETL and ELT pipelines using Databricks, PySpark, Spark SQL, Python, and SQL.
• Develop reusable ingestion and transformation frameworks for structured, semi-structured, and streaming data.
• Implement batch, incremental, change-data-capture, and streaming processing patterns.
• Build and maintain Delta Lake tables using medallion architecture across Bronze, Silver, and Gold layers.
• Develop dbt models, tests, macros, packages, documentation, and incremental processing patterns.
• Create and maintain Apache Airflow DAGs and Databricks Workflows with dependency management, retries, alerting, and operational controls.
• Integrate data from APIs, databases, files, event streams, and cloud data services.
• Produce technical designs, mapping specifications, lineage documentation, deployment instructions, and operational runbooks.
• Monitor pipeline health and resolve failures, performance degradation, data defects, and service-level breaches.
• Perform root-cause analysis and implement durable preventive measures.
• Support release readiness, production cutover, incident resolution, and ongoing platform operations.
• Improve compute utilization and cost efficiency across batch and streaming workloads.
• Collaborate with platform engineers to deploy data assets through Terraform and Databricks Asset Bundles where applicable.
• Follow enterprise architecture, security, data-governance, and regulatory requirements.
• Partner with architects, product owners, analysts, data scientists, governance teams, and business stakeholders.
• Translate business requirements into scalable data models, pipelines, and technical work packages.
• Conduct code reviews and enforce engineering, documentation, testing, and support standards.
• Mentor junior and mid-level engineers and share reusable patterns and best practices.
• Communicate delivery status, risks, dependencies, and technical trade-offs clearly.

Skills

Databricks

Apache Spark

PySpark

SQL

Python

dbt

Airflow

Delta Lake

Unity Catalog

CI/CD

Production support

Tools

Airflow

Databricks

dbt

Terraform

Git

Job description

Acestack in Irvine, CA is seeking a hands-on Senior Data Engineer to design, build, and optimize enterprise-scale data solutions on the Databricks Lakehouse Platform. You will develop batch and streaming pipelines, modernize ETL workloads, and deliver trusted data products for analytics and risk management.

The ideal candidate has deep experience with Databricks, Spark, PySpark, SQL, Python, dbt, Airflow, Delta Lake, and Unity Catalog, including production support and CI/CD. This role is onsite.