Databricks Data Engineer | Scalable Pipelines & Lakehouse

Xinova Group; New York (NY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Xinova Group is seeking a Databricks Data Engineer to design, build and optimise scalable data pipelines and lakehouse solutions for analytics, reporting and AI initiatives.

This hands-on role requires strong experience with Databricks, PySpark and SQL, and involves building ETL/ELT pipelines, Delta Lake tables across bronze, silver and gold layers, governance and CI/CD pipelines. Hybrid work in New York.

Qualifications

• Proficiency in Python, PySpark and SQL.
• Experience building ETL/ELT pipelines and working with large datasets.
• Knowledge of Apache Spark, Delta Lake and lakehouse architecture.
• Experience with Azure, AWS or GCP.
• Understanding of data modelling, warehousing and data quality practices.
• Troubleshooting distributed processing workloads and improving performance.
• Familiarity with Git, CI/CD and software engineering best practices.
• Strong communication and collaboration with technical and business teams.
• Ability to work on a hybrid basis in New York.

Responsibilities

• Design, develop and maintain data pipelines using Databricks, PySpark and SQL.
• Build batch and streaming workflows to ingest data from databases, APIs and enterprise applications.
• Develop and maintain Delta Lake tables across bronze, silver and gold layers.
• Implement data transformation, validation and quality controls.
• Optimise Spark workloads, cluster configurations and processing costs.
• Implement governance, access controls and data discovery through Unity Catalog.
• Troubleshoot pipeline failures and resolve performance and reliability issues.
• Build reusable engineering components and follow coding, testing and documentation standards.
• Support version control and automated deployments through Git and CI/CD pipelines.
• Collaborate with stakeholders to translate business requirements into technical solutions.

Skills

Python

PySpark

SQL

ETL/ELT

Apache Spark

Delta Lake

Lakehouse architecture

Git

CI/CD

Cloud platforms

Tools

Databricks

Unity Catalog

Lakeflow

Terraform

Job description

Xinova Group is seeking a Databricks Data Engineer to design, build and optimise scalable data pipelines and lakehouse solutions for analytics, reporting and AI initiatives.

This hands-on role requires strong experience with Databricks, PySpark and SQL, and involves building ETL/ELT pipelines, Delta Lake tables across bronze, silver and gold layers, governance and CI/CD pipelines. Hybrid work in New York.