Remote Databricks Engineer - PySpark & Lakehouse

Concentrix CVG Customer Management Group Inc.; Texas; Hybrid; USD 92,000 - 120,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

Comprehensive employee assistance

401(k) retirement plan

Paid time off and holidays

Paid learning days

Job summary

Concentrix CVG Corporation is seeking a Databricks Engineer to design and maintain data pipelines on Databricks across Azure, AWS, or GCP. You will leverage Python, PySpark, SQL, and Delta Lake to deliver robust ingestion, transformation, and modeling capabilities, collaborating with senior engineers to own end-to-end pipeline delivery.

You will work with cross-functional teams to meet performance goals, implement data quality checks, and participate in code reviews, CI/CD, and governance

Qualifications

• 6+ years of professional experience in data engineering, data & analytics or data warehousing.
• 3+ years of hands-on experience with Databricks in a data engineering capacity.
• 3+ years of experience with Python, PySpark DataFrames, Spark SQL, or pandas API on Spark.
• 2+ years of data engineering on Microsoft Azure (ADLS Gen2, Azure Data Factory, Key Vault).

Responsibilities

• Build and maintain batch and streaming data pipelines for ETL/ELT into the Databricks lakehouse on Azure.
• Implement data ingestion using Auto Loader, Lakeflow Connect, and Azure Data Factory.
• Develop pipelines with Lakeflow Spark Declarative Pipelines, including data quality expectations.
• Write PySpark and Spark SQL transformations to prepare data for analytics and reporting.
• Configure, schedule, and monitor data workflows with Lakeflow Jobs and Databricks Workflows.
• Apply medallion architecture and Unity Catalog permissions to data assets.
• Write unit tests and deploy pipelines with CI/CD using Git folders and Declarative Automation Bundles.
• Monitor, troubleshoot, and resolve pipeline failures and data quality issues in production.
• Collaborate with cross-functional teams to ensure data solutions meet business requirements.

Skills

Databricks

Python

Spark SQL

SQL

Azure

Git

CI/CD

Pandas API

Tools

Delta Lake

Unity Catalog

Lakeflow

Azure Data Factory

ADLS Gen2

Databricks SQL

Job description

Concentrix CVG Corporation is seeking a Databricks Engineer to design and maintain data pipelines on Databricks across Azure, AWS, or GCP. You will leverage Python, PySpark, SQL, and Delta Lake to deliver robust ingestion, transformation, and modeling capabilities, collaborating with senior engineers to own end-to-end pipeline delivery.

You will work with cross-functional teams to meet performance goals, implement data quality checks, and participate in code reviews, CI/CD, and governance