Databricks Data Engineer

Veriipro; United States; On-site; USD 120,000 - 170,000; Full time

Job summary

Veriipro seeks a Senior Databricks Data Engineer to design, develop and optimize data pipelines using Databricks, PySpark, and Delta Lake in a fast-paced setting. You will contribute to scalable data platforms, governance, automation, and CI/CD initiatives.

Ideal candidates have hands-on Databricks experience, PySpark and Delta Tables expertise, plus strong Python (OOP) and Git/GitHub skills within an Agile team.

Qualifications

• Strong hands-on Databricks Data Engineering experience.
• Expertise with PySpark, Delta Tables, and Declarative Pipelines.
• Experience with Databricks Asset Bundles and Unity Catalog governance.
• Proficiency in Python with OOP and clean code practices.
• Knowledge of CI/CD processes and automation.
• Familiarity with Git/GitHub and branching strategies.
• Experience in Agile development environments.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build and optimize Delta Lake solutions using Delta Tables and Declarative Pipelines.
• Implement Databricks Asset Bundles for deployment and lifecycle management.
• Manage data governance, security, and permissions using Unity Catalog.
• Configure cluster policies and platform governance standards.
• Develop clean, maintainable, and reusable Python code.
• Implement CI/CD workflows using Git/GitHub best practices.
• Collaborate with cross-functional teams in an Agile environment.
• Optimize data processing performance and troubleshoot data engineering issues.
• Support automation initiatives using Terraform and Bash scripting.

Skills

Databricks

PySpark

Delta Tables

Declarative Pipelines

Unity Catalog

Python (OOP)

CI/CD

Git/GitHub

Agile development

SQL

Tools

Terraform

Bash scripting

GitHub Actions

Job description

We are seeking a Senior Databricks Data Engineer with strong experience designing, developing, and optimizing data engineering solutions using Databricks, PySpark, Delta Lake, and modern data engineering practices. The ideal candidate will have expertise in data pipelines, cloud data platforms, governance, automation, and CI/CD processes.

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build and optimize Delta Lake solutions using Delta Tables and Declarative Pipelines.
• Implement Databricks Asset Bundles for deployment and lifecycle management.
• Manage data governance, security, and permissions using Unity Catalog.
• Configure and enforce cluster policies and platform governance standards.
• Develop clean, maintainable, and reusable Python code.
• Implement CI/CD workflows using Git/GitHub best practices.
• Collaborate with cross-functional teams in an Agile environment.
• Optimize data processing performance and troubleshoot data engineering issues.
• Support automation initiatives using tools such as Terraform and Bash scripting.

Required Technical Skills

• Strong hands-on experience with Databricks Data Engineering.
• Expertise in PySpark, Delta Tables, and Declarative Pipelines.
• Experience with Databricks Asset Bundles (DAB).
• Strong knowledge of Unity Catalog, access controls, and permission management.
• Experience with cluster policies, governance, and Databricks administration best practices.
• Strong Python development skills with understanding of Object-Oriented Programming (OOP).
• Knowledge of clean code principles and software development best practices.
• Experience with CI/CD processes and automation.
• Strong understanding of Git/GitHub, including pull requests and branching strategies.
• Experience working in Agile development environments.

Preferred Skills

• Strong SQL skills.
• Experience with Terraform for infrastructure automation.
• Familiarity with Bash scripting and command-line tools.