Senior Solutions Consultant

Unison Group; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Unison Group seeks a Senior Consultant – Databricks to design, implement, and optimize data engineering solutions on the Databricks platform. You will build scalable pipelines, ensure data quality, and collaborate with customers and internal teams to deliver robust data lakehouse solutions.

The role requires hands-on experience with Databricks, Apache Spark, Python, and SQL, and a strong ability to translate business requirements into scalable data solutions within a cloud-based Lakehouse

Qualifications

• 6+ years in Information Technology.
• 3+ years hands-on with Databricks.
• Experience in enterprise data engineering solutions.

Responsibilities

• Understand data and analytics requirements with customers and stakeholders.
• Design, develop, and maintain scalable data pipelines with Databricks and Spark.
• Build and optimize ETL/ELT workflows for batch and near real-time processing.
• Implement Data Lakehouse solutions using Bronze, Silver, Gold architecture.
• Develop data transformations with Python, PySpark, and SQL.
• Ensure data quality, reliability, and performance via testing and tuning.
• Troubleshoot data processing and performance issues.
• Collaborate with Solution Architects, Data Engineers, and DevOps to deliver end-to-end solutions.
• Participate in code reviews and follow best practices.
• Create technical docs and assist deployment/production activities.

Skills

Databricks

Python

SQL

Apache Spark

ETL/ELT design

Data Lakehouse

Data modeling

Tools

Azure Data Factory

Airflow

MLflow

Git

CI/CD

Job description

Job Summary

We are seeking a skilled Senior Consultant – Databricks to design, develop, and implement modern data engineering solutions using the Databricks platform. The ideal candidate will work closely with customers and internal teams to understand business requirements, build scalable data pipelines, and deliver high-quality data solutions leveraging cloud-based Lakehouse architecture.

The role requires strong hands-on experience in Databricks, Apache Spark, Python, and SQL, along with the ability to collaborate with stakeholders and contribute to end-to-end data platform implementations.

Experience

• 6+ years of overall experience in Information Technology.


• 3+ years of hands-on experience with Databricks.


• Experience in developing and implementing enterprise data engineering solutions.

Key Responsibilities

• Collaborate with customers and business stakeholders to understand data and analytics requirements.


• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.


• Build and optimize ETL/ELT workflows for batch and near real-time data processing.


• Implement Data Lakehouse solutions following the Medallion Architecture (Bronze, Silver, Gold).


• Develop data transformation logic using Python, PySpark, and SQL.


• Ensure data quality, reliability, and performance through testing and optimization.


• Troubleshoot and resolve data processing and performance issues.


• Work closely with Solution Architects, Data Engineers, and DevOps teams to deliver end-to-end solutions.


• Participate in code reviews and follow development best practices.


• Create technical documentation and provide support during deployment and production activities.

Required Technical Skills

• Strong hands-on experience with Databricks.


• Proficiency in Python, SQL, and Apache Spark (PySpark).


• Good understanding of Data Lake and Lakehouse architecture.


• Experience implementing Bronze, Silver, and Gold data layers.


• Experience designing and developing ETL/ELT pipelines.


• Knowledge of Delta Lake and Spark performance optimization.


• Understanding of data modeling and data integration concepts.


• Experience working with structured and semi-structured data.

Preferred Skills

• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform.


• Familiarity with Delta Live Tables, Unity Catalog, Databricks Workflows, and MLflow.


• Experience with orchestration tools such as Azure Data Factory, Apache Airflow, or similar.


• Knowledge of Git, CI/CD pipelines, and Agile development methodologies.


• Databricks or cloud platform certifications are an added advantage.

Soft Skills

• Strong analytical and problem-solving skills.


• Good communication and customer interaction skills.


• Ability to gather and analyze business requirements.


• Strong collaboration and teamwork skills.


• Ability to manage multiple tasks and deliver within project timelines.


• Willingness to learn new technologies and adapt to evolving project requirements.

What You'll Bring

• Hands-on expertise in Databricks and modern data engineering practices.


• A customer-focused approach with the ability to translate business requirements into technical solutions.


• A passion for building scalable, reliable, and high-performance data platforms.


• The ability to work independently as well as collaboratively in cross-functional teams.