Senior Python PySpark Data Engineer – Databricks

Compunnel, Inc.; Mississauga; Remote; CAD 90,000 - 120,000; Full time

Job summary

A tech solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills in Python, PySpark, and Azure DevOps, with over 10 years of software development experience. Familiarity with Agile methodologies and CI/CD practices is essential. The position supports remote work and offers opportunities for collaboration and innovation.

Qualifications

• Over 10 years of software development experience.
• Minimum 6 years of hands-on experience with Python.
• Strong expertise in PySpark and Databricks.
• Proficient in Azure DevOps and CI/CD practices.
• Experience with Python testing tools like Pytest and Pylint.
• Solid understanding of algorithms and software architectures.

Responsibilities

• Develop and maintain applications using Python and PySpark.
• Implement ETL/ELT processes and integrate data pipelines.
• Utilize Databricks for processing and analytics.
• Perform code testing using Pytest and Pylint.
• Collaborate with cross-functional teams to deliver solutions.

Skills

Python

PySpark

Databricks

Azure DevOps

ETL/ELT processes

APIs

Tools

Pytest

Pylint

Django

Flask

Jira

Job description

A tech solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills in Python, PySpark, and Azure DevOps, with over 10 years of software development experience. Familiarity with Agile methodologies and CI/CD practices is essential. The position supports remote work and offers opportunities for collaboration and innovation.