Databricks Developer

KData AI; Montreal (administrative region); Hybrid; CAD 80,000 - 120,000; Full time

Job summary

KData AI is seeking a Senior Data Engineer located in Quebec. The role involves designing and developing robust enterprise applications in Java and Angular, along with leveraging Databricks for automation. The candidate must have a minimum of 6 years of experience and excellent knowledge of data engineering best practices.

This hybrid role requires working at the Montreal office twice a week, depending on client demand. Strong skills in Python and SQL are essential, alongside familiarity with MLOps tools and the Agile process.

Qualifications

• Minimum of 6 years related experience, senior resource.
• Strong knowledge of Azure DevOps for source code management.
• Good understanding of distributed data processing concepts.

Responsibilities

• Design and develop applications in Java and Angular.
• Develop reusable components and libraries.
• Take charge of end-to-end delivery from design to deployment.
• Leverage Databricks for automation and data validation.

Skills

Data engineering best practices

Azure Databricks

Python

SQL

ETL concepts

Spark

MLOps tools

Agile process

JIRA

Confluence

Job description

Responsibilities

• Design and develop robust enterprise applications that impact operations in Java (backend) and Angular (frontend).
• Develop reusable components, libraries and utilities to accelerate solution development within the team.
• Take charge of end-to-end deliveries: from understanding the need to production deployment, including design, development, testing, documentation and support.
• Leverage the data available on Databricks and develop Python and SQL scripts for automation, prototyping, and data validation.
• Analyze and understand the business needs of different stakeholders in order to propose appropriate application and analytical solutions.
• Collaborate closely with business teams, data analysts and other developers to ensure the relevance and quality of the solutions delivered.

Requirements

• Minimum of 6 years of experience related to the mandate, senior resource.
• Very good knowledge of best practices in data engineering and data modeling.
• Very good knowledge of the Azure Databricks and Azure Data Factory platform, their functionalities, including Azure DevOps for source code management and branching.
• Very good knowledge of the Python programming language and SQL.
• Good understanding of the ETL concept, medallion architecture and processes related to data ingestion and exposure in the context of cloud analytics solutions.
• Good understanding of distributed data processing concepts, Spark and big data.
• Good knowledge of Python libraries, how to use them and how to create new ones with a view to reuse.
• Knowledge of MLOps tools, the artificial intelligence solutions lifecycle and related concepts.
• Knowledge of the Agile process within the framework of a DevOps working group and of JIRA and Confluence software.
• Experience with VS Code or another text editor.
• The candidate must be located in Quebec and be able to work in a hybrid model (twice a week at the Montreal office, depending on client demand).

The resource must be located in Quebec.