Databricks Engineer

Xinova Group; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

Xinova Group is seeking an experienced Databricks Engineer to support a Fortune 500 utilities customer's platform modernization initiative. You will design, develop, and optimize data pipelines, work with Spark, PySpark, Python, SQL, and Delta Lake, and help build a unified Lakehouse for analytics and operations.

You will collaborate with data architects, engineers, and business stakeholders to deliver scalable, enterprise-grade data solutions across cloud environments, ensuring data quality,

Qualifications

• 5+ years of experience in Data Engineering, Software Engineering, or a related technical discipline.
• 2+ years hands-on experience developing solutions using Databricks.
• Strong proficiency in Python, PySpark, and SQL.
• Extensive experience with Apache Spark and distributed data processing.
• Proven experience designing, developing, and maintaining production-grade ETL/ELT data pipelines.
• Strong understanding of Databricks Lakehouse architecture and Delta Lake.
• Experience implementing data ingestion, transformation, and processing frameworks across complex enterprise environments.
• Hands-on experience with Delta Lake, including Delta tables, schema evolution, and optimization techniques.
• Experience working with cloud platforms, particularly Microsoft Azure, AWS, or Google Cloud Platform.
• Strong understanding of data warehousing, data modelling, and modern data architecture principles.
• Experience developing batch processing and, ideally, real-time streaming solutions.
• Familiarity with data orchestration, workflow automation, and production pipeline monitoring.
• Understanding of data governance, security, access controls, and data quality frameworks.
• Experience with Git, CI/CD pipelines, and modern software development practices.
• Experience integrating data from multiple enterprise applications and legacy systems.
• Understanding of customer data structures, customer information management, and enterprise data integration.
• Strong analytical and problem-solving skills, with the ability to troubleshoot complex engineering challenges.
• Excellent communication skills and the ability to collaborate effectively with technical and non-technical stakeholders.

Responsibilities

• Design, develop, and maintain enterprise-scale data engineering solutions using Databricks Lakehouse Platform.
• Support modernization of the customer data platform by migrating legacy data infrastructure into Databricks.
• Build and optimize complex ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake.
• Develop scalable data ingestion frameworks to integrate customer information from multiple systems.
• Consolidate and transform customer data from disparate legacy platforms into a unified data environment.
• Design and implement data models supporting customer profiles, accounts, service interactions, billing, and lifecycle management.
• Develop robust data transformation, cleansing, validation, and enrichment processes to improve data quality.
• Design and develop Lakehouse architectures utilizing Delta Lake and Bronze/Silver/Gold data layers.
• Build and maintain batch and real-time streaming data pipelines using Spark and Databricks.

Skills

Databricks engineering

Collaboration

Problem solving

Communication

Tools

Databricks

Apache Spark

PySpark

Python

SQL

Delta Lake

Azure

AWS

GCP

Job description

A Fortune 500 utilities organization is seeking an experienced Databricks Engineer to support a major Customer Platform Modernization initiative built around the Databricks Lakehouse Platform.

This is an exciting opportunity to contribute to a significant enterprise data transformation, modernizing the organization's customer data infrastructure and developing scalable data solutions to support improved customer experiences, operational efficiency, and advanced analytics.

The successful candidate will play a hands‑on role in designing, developing, and optimizing data pipelines and integration frameworks that bring together customer information from multiple legacy and enterprise systems. Working closely with Data Architects, Data Engineers, Enterprise Technology teams, and business stakeholders, this individual will help establish a modern, unified data environment capable of supporting the organization's evolving customer platform requirements.

We are specifically seeking a hands‑on Databricks Engineer with strong expertise in Apache Spark, PySpark, Python, SQL, and Delta Lake. The ideal candidate will have experience developing production-grade data pipelines, integrating complex enterprise systems, and delivering scalable Lakehouse solutions within large-scale cloud environments.

Key Responsibilities

• Design, develop, and maintain enterprise-scale data engineering solutions using the Databricks Lakehouse Platform
• Support the modernization of the organization's existing customer data platform, migrating legacy data infrastructure and integration processes into Databricks
• Build and optimize complex ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake
• Develop scalable data ingestion frameworks to integrate customer information from multiple internal and external enterprise systems
• Consolidate and transform customer data from disparate legacy platforms into a consistent, unified data environment
• Design and implement data models supporting customer profiles, account information, service interactions, billing, usage, and customer lifecycle management
• Develop robust data transformation, cleansing, validation, and enrichment processes to improve customer data quality and consistency
• Design and develop Lakehouse architectures utilizing Delta Lake and Bronze, Silver, and Gold data layers
• Build and maintain batch and real-time streaming data pipelines using Apache Spark and Databricks technologies
• Develop and optimize Delta Lake tables, including partitioning, schema evolution, and performance tuning strategies
• Implement and manage data workflows using Databricks Workflows, Delta Live Tables, and related orchestration technologies
• Work with Databricks Unity Catalog to establish data governance, access controls, security, and metadata management
• Integrate Databricks with existing customer platforms, CRM systems, billing applications, enterprise applications, and downstream data consumers
• Optimize Spark jobs, SQL queries, and data processing workloads to improve platform performance, scalability, and cloud cost efficiency
• Develop reusable data engineering frameworks, libraries, and coding standards to improve development efficiency
• Implement CI/CD pipelines, version control, and Infrastructure-as-Code practices to support reliable platform deployments
• Collaborate with Data Architects, Analytics teams, and business stakeholders to translate customer platform requirements into scalable technical solutions
• Ensure customer data solutions meet enterprise security, privacy, regulatory, governance, and data quality requirements
• Troubleshoot complex data pipeline failures, performance bottlenecks, and integration challenges across enterprise environments
• Support production deployments, platform enhancements, and ongoing operational improvements
• Contribute to the development of a modern customer data foundation supporting advanced analytics, customer insights, personalization, and AI-driven initiatives

Required Qualifications

• 5+ years of experience in Data Engineering, Software Engineering, or a related technical discipline
• 2+ years of hands‑on experience developing solutions using Databricks
• Strong proficiency in Python, PySpark, and SQL
• Extensive experience with Apache Spark and distributed data processing
• Proven experience designing, developing, and maintaining production‑grade ETL/ELT data pipelines
• Strong understanding of Databricks Lakehouse architecture and Delta Lake
• Experience implementing data ingestion, transformation, and processing frameworks across complex enterprise environments
• Hands‑on experience with Delta Lake, including Delta tables, schema evolution, and optimization techniques
• Experience working with cloud platforms, particularly Microsoft Azure, AWS, or Google Cloud Platform
• Strong understanding of data warehousing, data modelling, and modern data architecture principles
• Experience developing batch processing and, ideally, real‑time streaming solutions
• Familiarity with data orchestration, workflow automation, and production pipeline monitoring
• Understanding of data governance, security, access controls, and data quality frameworks
• Experience with Git, CI/CD pipelines, and modern software development practices
• Experience integrating data from multiple enterprise applications and legacy systems
• Understanding of customer data structures, customer information management, and enterprise data integration
• Strong analytical and problem‑solving skills, with the ability to troubleshoot complex engineering challenges
• Excellent communication skills and the ability to collaborate effectively with technical and non‑technical stakeholders

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional certification
• Experience with Databricks Unity Catalog, Delta Live Tables, and Databricks Workflows
• Experience implementing Medallion Architecture and enterprise Lakehouse solutions
• Knowledge of Databricks Asset Bundles and automated deployment frameworks
• Experience with Databricks SQL and developing optimized analytical workloads
• Familiarity with MLflow, Feature Store, and machine learning operationalization
• Experience supporting Generative AI, Large Language Models (LLMs), and AI‑driven data solutions
• Knowledge of cloud‑native data services, including Azure Data Factory, AWS Glue, S3, ADLS, and related technologies
• Experience with Terraform, Infrastructure‑as‑Code, and cloud platform automation
• Familiarity with dbt, Airflow, Kafka, Fivetran, or other modern data engineering technologies
• Experience migrating legacy customer platforms, data warehouses, CRM environments, or traditional ETL frameworks to Databricks
• Experience integrating CRM, customer billing, customer relationship management, and enterprise operational systems
• Previous experience within the utilities, energy, power generation, transmission, distribution, or wider infrastructure sectors
• Understanding of utilities customer data domains, including customer accounts, service connections, metering, billing, energy consumption, and customer interactions
• Experience working within Fortune 500 organizations or other highly complex enterprise environments

Project Context

The organization is undertaking a significant Customer Platform Modernization initiative, leveraging Databricks to transform how customer data is integrated, managed, and made available across the enterprise.

As a Fortune 500 utilities business, the organization manages substantial volumes of customer information across multiple legacy platforms, billing systems, customer applications, and operational databases.

The project is focused on modernizing this existing customer data infrastructure by bringing disparate data sources together within Databricks, establishing consistent data models, and developing scalable ingestion and transformation pipelines.

The objective is to create a more unified and reliable customer data foundation that supports improved customer visibility, more efficient business processes, and enhanced analytical capabilities.

The successful candidate will contribute to building and optimizing this modern data environment, working across the full data engineering lifecycle from ingestion and transformation through to data modelling, governance, and downstream consumption.

Key areas of focus include:

• Customer data integration and consolidation across legacy enterprise systems
• Modernization of existing customer data pipelines and ETL processes
• Development of scalable customer data models and Lakehouse architecture
• Improving customer data quality, consistency, and accessibility
• Supporting customer analytics, reporting, and operational applications
• Establishing a reliable data foundation for future digital transformation and AI initiatives

Previous utilities or energy sector experience is advantageous but not essential. The priority is demonstrable Databricks engineering capability, strong technical fundamentals, and experience delivering enterprise‑scale data modernization solutions.

What We're Looking For

We are seeking a hands‑on Databricks Engineer who combines strong technical expertise with practical experience delivering enterprise data engineering and modernization projects.

The ideal candidate will have a proven track record of building production‑grade pipelines, integrating complex enterprise systems, developing robust data models, and optimizing distributed data processing workloads.

You should be comfortable working within a large enterprise technology environment, collaborating with multiple engineering teams, and translating business requirements into scalable technical solutions.

We are particularly interested in engineers who understand Databricks beyond simply using it as a development environment and can demonstrate genuine experience building, optimizing, and supporting solutions on the platform.

Candidates with strong PySpark, Python, SQL, Spark, and Delta Lake experience, alongside exposure to customer data platforms, enterprise integrations, and legacy modernization initiatives, will be particularly relevant.

Experience within utilities, energy, or other highly regulated infrastructure environments would be beneficial, particularly where candidates have worked on large‑scale customer data or platform transformation projects.