Senior Databricks Data Engineer - Remote

Concentrix; Austin (TX); Remote; USD 106,000 - 140,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

401(k) retirement plan

Paid time off and holidays

Paid learning days

Job summary

Concentrix is seeking a Senior Databricks Engineer to own end-to-end data pipelines on the Databricks lakehouse across Azure, AWS, or GCP. You will lead solution design, set engineering standards, and mentor other engineers while leveraging Python, PySpark, and SQL to enable high-value analytics and AI/ML use cases.

The role emphasizes data governance, CI/CD for pipelines, and collaboration with business stakeholders to meet performance and reliability goals.

Qualifications

• 10+ years of professional data engineering experience.
• 5+ years with Databricks in a data engineering capacity.
• 3+ years of data engineering on Microsoft Azure.
• Strong Python, PySpark, Spark SQL skills.
• Experience with data governance and Delta Lake.

Responsibilities

• Design, develop, and maintain robust batch and streaming data pipelines for ETL/ELT into the Databricks lakehouse on Azure.
• Automate data ingestion using Auto Loader, Lakeflow, and Azure services like Data Factory and Event Hubs.
• Build and manage pipelines with Lakeflow Spark Declarative Pipelines and define data quality expectations.
• Perform PySpark and Spark SQL transformations for analytics, reporting, and AI/ML use cases.
• Design scalable data models using medallion architecture (bronze/silver/gold) and SCD types.
• Implement Unity Catalog governance, access controls, lineage, and data masking where needed.
• Optimize Delta tables and Spark workloads for cost and performance across environments.
• Establish CI/CD and testing for pipelines with Git, Databricks Bundles, and Azure DevOps/GitHub Actions.
• Monitor pipeline health and lead root-cause analysis for production issues.
• Lead design reviews, set engineering standards, and mentor data engineers.
• Collaborate with stakeholders to ensure solutions meet business goals and performance needs.

Skills

Databricks

Python

PySpark

SQL

Data Modeling

Data Governance

Education

Bachelor's degree in computer science, Information Systems, Data Engineering, or related

Tools

Azure

AWS

GCP

Azure Data Factory

Lakeflow

Unity Catalog

Delta Lake

Power BI

Terraform

Job description

Concentrix is seeking a Senior Databricks Engineer to own end-to-end data pipelines on the Databricks lakehouse across Azure, AWS, or GCP. You will lead solution design, set engineering standards, and mentor other engineers while leveraging Python, PySpark, and SQL to enable high-value analytics and AI/ML use cases.

The role emphasizes data governance, CI/CD for pipelines, and collaboration with business stakeholders to meet performance and reliability goals.