Senior Databricks Developer - Spark, Delta Lake & AI

Cognizant; Mississauga; Hybrid; CAD 90,000 - 120,000; Full time

Job summary

A leading technology services company is seeking a skilled professional in Artificial Intelligence and Analytics to develop scalable data processing solutions using Spark in Scala. The role involves collaborating with teams, mentoring junior developers, and utilizing Databricks technologies. Candidates should possess strong backgrounds in Python, Databricks SQL, and Delta Lake while being able to work effectively in a hybrid model. This is an excellent opportunity for those who are passionate about data-driven decision-making and innovation.

Qualifications

• Strong background in Spark in Scala and Databricks technologies.
• Proficiency in Python programming for data processing tasks.
• Experience with Databricks Unity Catalog Admin and Databricks CLI.
• Expertise in Databricks SQL and Delta Lake for data management.
• Knowledge of PySpark for efficient data processing.
• Ability to work in a hybrid model.
• Excellent problem-solving skills and attention to detail.

Responsibilities

• Develop and implement scalable data processing solutions using Spark in Scala.
• Administer and manage Databricks Unity Catalog for data governance.
• Utilize Databricks CLI to automate workflows.
• Write Python scripts for data integration and transformation.
• Design Databricks SQL queries for improved reporting efficiency.
• Implement Databricks Delta Lake for data reliability.
• Create Databricks Workflows for automating data pipelines.
• Collaborate with teams to translate business requirements into solutions.
• Troubleshoot complex data processing issues.
• Provide mentorship to junior developers.
• Stay updated with industry trends.
• Ensure alignment with company standards.

Skills

Spark in Scala

Databricks technologies

Python programming

Databricks SQL

Delta Lake

PySpark

Problem-solving

Attention to detail

Job description

A leading technology services company is seeking a skilled professional in Artificial Intelligence and Analytics to develop scalable data processing solutions using Spark in Scala. The role involves collaborating with teams, mentoring junior developers, and utilizing Databricks technologies. Candidates should possess strong backgrounds in Python, Databricks SQL, and Delta Lake while being able to work effectively in a hybrid model. This is an excellent opportunity for those who are passionate about data-driven decision-making and innovation.