Databricks Data Engineer - Delta Lake, PySpark & CI/CD Pro

OpenTalent; Raritan (NJ); On-site; USD 100,000 - 180,000; Full time

Job summary

OpenTalent in New Jersey is seeking a Data Engineer with Databricks expertise to design, build, and maintain scalable ETL/ELT pipelines using notebooks and PySpark.

You will implement Unity Catalog, Delta Sharing, and data governance while collaborating with cross-functional teams to deliver robust data solutions and ensure performance. Strong Python, SQL, and DevOps skills are required, with experience in Medallion Architecture and data warehousing.

Qualifications

• Hands-on experience building scalable ETL/ELT pipelines using Databricks and notebooks.
• Strong Python and PySpark skills for large-scale data processing.
• Experience with Delta Lake, Delta Sharing, and Unity Catalog for data governance.
• Experience implementing CI/CD for Databricks workloads and DevOps pipelines.
• Familiarity with Medallion Architecture (Bronze/Silver/Gold) and data modeling concepts.
• Proficient in SQL and query optimization; ability to design scalable data warehouses.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines and notebooks using Databricks.
• Build and optimize data processing solutions with PySpark and Python.
• Implement and manage data governance, security, and access controls with Unity Catalog.
• Leverage Delta Lake features including Delta Sharing and Time Travel; optimize data.
• Develop and maintain CI/CD pipelines for Databricks deployments using industry-standard DevOps practices.
• Perform performance tuning and optimization of Databricks workloads.
• Collaborate with architects, BAs, and stakeholders to translate data requirements into solutions.
• Support production deployments, troubleshooting, monitoring, and continuous improvements.
• Adhere to coding standards, documentation, and SDLC best practices.

Skills

Databricks

PySpark

Python

Delta Lake

Delta Sharing

Unity Catalog

Structured Streaming

Workflow Orchestration

Data Governance

SQL

CI/CD

DevOps

Medallion Architecture

Data Modeling

Data Warehousing

Data Quality

Observability

Tools

Git

Job description

OpenTalent in New Jersey is seeking a Data Engineer with Databricks expertise to design, build, and maintain scalable ETL/ELT pipelines using notebooks and PySpark.

You will implement Unity Catalog, Delta Sharing, and data governance while collaborating with cross-functional teams to deliver robust data solutions and ensure performance. Strong Python, SQL, and DevOps skills are required, with experience in Medallion Architecture and data warehousing.