Databricks Engineer

Xinova Group; New York (NY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Xinova Group is seeking a Databricks Data Engineer to design, build and optimise scalable data pipelines and lakehouse solutions for analytics, reporting and AI initiatives.

This hands-on role requires strong experience with Databricks, PySpark and SQL, and involves building ETL/ELT pipelines, Delta Lake tables across bronze, silver and gold layers, governance and CI/CD pipelines. Hybrid work in New York.

Qualifications

• Proficiency in Python, PySpark and SQL.
• Experience building ETL/ELT pipelines and working with large datasets.
• Knowledge of Apache Spark, Delta Lake and lakehouse architecture.
• Experience with Azure, AWS or GCP.
• Understanding of data modelling, warehousing and data quality practices.
• Troubleshooting distributed processing workloads and improving performance.
• Familiarity with Git, CI/CD and software engineering best practices.
• Strong communication and collaboration with technical and business teams.
• Ability to work on a hybrid basis in New York.

Responsibilities

• Design, develop and maintain data pipelines using Databricks, PySpark and SQL.
• Build batch and streaming workflows to ingest data from databases, APIs and enterprise applications.
• Develop and maintain Delta Lake tables across bronze, silver and gold layers.
• Implement data transformation, validation and quality controls.
• Optimise Spark workloads, cluster configurations and processing costs.
• Implement governance, access controls and data discovery through Unity Catalog.
• Troubleshoot pipeline failures and resolve performance and reliability issues.
• Build reusable engineering components and follow coding, testing and documentation standards.
• Support version control and automated deployments through Git and CI/CD pipelines.
• Collaborate with stakeholders to translate business requirements into technical solutions.

Skills

Python

PySpark

SQL

ETL/ELT

Apache Spark

Delta Lake

Lakehouse architecture

Git

CI/CD

Cloud platforms

Tools

Databricks

Unity Catalog

Lakeflow

Terraform

Job description

We are seeking a Databricks Data Engineer to design, build and optimise scalable data pipelines and lakehouse solutions. You will work closely with data architects, analysts and business stakeholders to deliver reliable, high-quality data for analytics, reporting and AI initiatives.

This is a hands-on engineering role requiring strong experience with Databricks, PySpark and SQL.

Key Responsibilities

• Design, develop and maintain data pipelines using Databricks, PySpark and SQL.
• Build batch and streaming workflows to ingest data from databases, APIs and enterprise applications.
• Develop and maintain Delta Lake tables across bronze, silver and gold layers.
• Implement data transformation, validation and quality controls to ensure accurate, reliable datasets.
• Optimise Spark workloads, cluster configurations and processing costs.
• Implement governance, access controls and data discovery through Unity Catalog.
• Troubleshoot pipeline failures and resolve performance and reliability issues.
• Build reusable engineering components and follow coding, testing and documentation standards.
• Support version control and automated deployments through Git and CI/CD pipelines.
• Collaborate with stakeholders to translate business requirements into technical solutions.

Required Experience

• Proficiency in Python, PySpark and SQL.
• Experience building ETL/ELT pipelines and working with large datasets.
• Practical knowledge of Apache Spark, Delta Lake and lakehouse architecture.
• Experience with at least one major cloud platform: Azure, AWS or GCP.
• Understanding of data modelling, warehousing and data quality practices.
• Experience troubleshooting distributed processing workloads and improving performance.
• Familiarity with Git, CI/CD and software engineering best practices.
• Strong communication skills and the ability to collaborate with technical and business teams.
• Ability to work on a hybrid basis in New York.

Preferred Experience

• Databricks certification, such as Data Engineer Associate or Professional.
• Experience with Unity Catalog, Lakeflow Jobs and Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables).
• Experience with streaming, change data capture and incremental processing.
• Familiarity with Terraform or other infrastructure-as-code tools.
• Experience supporting data platforms used for analytics, machine learning or AI.