Data Integration Engineer

Compunnel, Inc.; Halifax; Remote; CAD 60,000 - 80,000; Full time

Job summary

An innovative firm is on the lookout for a skilled Data Integration Engineer to craft and optimize data solutions using cutting-edge technologies like Azure SQL and Azure Databricks. This pivotal role focuses on building efficient data processing pipelines while ensuring data integrity and security. You'll collaborate with diverse teams to enhance performance and efficiency, making a significant impact on data management strategies. If you're passionate about data engineering and eager to work in a dynamic environment, this opportunity is perfect for you.

Qualifications

• 5+ years of hands-on experience with PySpark for data integration.
• Strong experience with Databricks and SQL for developing data solutions.

Responsibilities

• Implement scalable data solutions using Azure SQL and Databricks.
• Develop and deploy data processing pipelines for enhanced performance.

Skills

PySpark

Python

SQL

Tools

Azure SQL

Azure Databricks

Job description

We are seeking an experienced Data Integration Engineer to design, develop, and optimize scalable data solutions using Azure SQL and Azure Databricks. This role involves implementing data processing pipelines, ensuring data integrity and security, and collaborating with cross-functional teams to enhance performance and efficiency.

Key Responsibilities

1. Implement scalable data solutions using Azure SQL and Azure Databricks
2. Develop and deploy data processing pipelines using Azure Databricks
3. Optimize data processing and storage for performance and efficiency
4. Collaborate with cross-functional teams to ensure data integrity and security

Required Qualifications

1. Minimum 5 years of hands-on experience with PySpark
2. Strong experience with Databricks, Python, and SQL
3. Strong background as an ETL developer

Location: Halifax, Nova Scotia - Remote