Databricks Lakehouse Lead Engineer

CNX; United States; Remote; USD 106,000 - 152,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401(k) Retirement Plan

Paid Time Off

Paid Learning Days

Job summary

Concentrix is hiring a Senior Databricks Engineer to own end-to-end data lifecycle across Azure/AWS/GCP, building scalable ingestion, transformation, and governance solutions using Databricks, Python, PySpark, and SQL.

You will lead design standards, mentor engineers, and collaborate with business and IT stakeholders to deliver production-ready data pipelines for analytics and AI/ML use cases.

Qualifications

• 10+ years of professional experience in data engineering, data and analytics, or data warehousing.
• 5+ years of hands-on experience working with Databricks in a data engineering capacity, including production workloads.
• 5+ years of experience with Python, including manipulating large datasets using PySpark DataFrames, Spark SQL, and/or the pandas API on Spark.
• 3+ years of data engineering experience on Microsoft Azure, including services such as ADLS Gen2, Azure Data Factory, Key Vault, and Microsoft Entra ID.
• Strong experience designing and developing ETL/ELT pipelines for batch and streaming data workloads.
• Experience with Databricks lakehouse architecture, Delta tables, Unity Catalog, Lakeflow Jobs, and Lakeflow Spark Declarative Pipelines.
• Experience implementing scalable data models using medallion architecture, dimensional modeling, and slowly changing dimensions.
• Experience with CI/CD and testing practices for data pipelines using Azure DevOps or GitHub Actions.
• Ability to lead technical design discussions, conduct code reviews, define engineering standards, and mentor other engineers.
• Strong collaboration skills with the ability to work across business, IT, and technical stakeholder groups.
• Preferred certification: Databricks Data Engineer Professional.

Responsibilities

• Design, develop, and maintain robust batch and streaming data pipelines for extracting, transforming, and loading data from multiple sources into the Databricks lakehouse on Azure.
• Automate and streamline data ingestion using Auto Loader, Lakeflow Connect, and Azure services such as Azure Data Factory and Event Hubs.
• Build and manage pipelines with Lakeflow Spark Declarative Pipelines, formerly Delta Live Tables, including data quality expectations and change data capture.
• Perform complex data manipulation and transformation using PySpark and Spark SQL to prepare data for analytics, reporting, and AI/ML use cases.
• Design and implement workflow orchestration and scheduling using Lakeflow Jobs, formerly Databricks Workflows, to support reliable data operations.
• Create scalable lakehouse data models using medallion architecture — bronze, silver, and gold — and dimensional modeling, including SCD Type 1 and Type 2.
• Implement data governance and security in Unity Catalog, including access controls, lineage, row filters, and column masks for sensitive data.
• Optimize performance and cost across Delta tables, Spark workloads, serverless and classic compute, and SQL warehouses.
• Establish CI/CD and testing practices for data pipelines using Git folders, Declarative Automation Bundles, formerly Databricks Asset Bundles, and Azure DevOps or GitHub Actions.
• Monitor pipeline health and data quality using system tables and data quality monitoring; lead root-cause analysis and resolution of production issues.
• Lead design and code reviews, define engineering standards, and mentor data engineers across the team.
• Collaborate with cross-functional teams, including business stakeholders and IT, to ensure data solutions meet business, performance, and project requirements.

Skills

Databricks

Python

PySpark

SQL

Azure

AWS

GCP

Delta Lake

ETL/ELT

CI/CD

Mentoring

Team leadership

Tools

Azure Data Factory

Key Vault

ADLS Gen2

Lakeflow

GitHub Actions

Azure DevOps

Job description

Concentrix is hiring a Senior Databricks Engineer to own end-to-end data lifecycle across Azure/AWS/GCP, building scalable ingestion, transformation, and governance solutions using Databricks, Python, PySpark, and SQL.

You will lead design standards, mentor engineers, and collaborate with business and IT stakeholders to deliver production-ready data pipelines for analytics and AI/ML use cases.