11737

Maarut; Toronto; On-site; CAD 120,000 - 180,000; Full time

Job summary

Maarut seeks a senior data engineer to design and implement large-scale data pipelines across cloud platforms. You will work on Databricks, DLT, Azure Data Factory, and data lakehouse architectures to move data from source systems to end users.

You will collaborate with data scientists and product teams, embrace DataOPS, CI/CD, and agile practices, and help optimize data workflows, performance tuning, and data security.

Qualifications

• Experience with designing and building complex data ETL/ELT pipelines.
• Experience with unified data platforms using Databricks and Unity Catalog.
• Strong SQL and data modeling skills; relational and dimensional modeling.
• Big data processing with PySpark and cloud data platforms.
• Azure Data Factory pipelines and CI/CD integration.
• Data migration across Oracle, Azure Data Lake, Databricks, and Fabric.

Responsibilities

• Design, build, automate and optimize complex data ETL/ELT processes using Databricks, Delta Live Tables (DLT) and equivalent.
• Develop unified data platforms using Databricks and Unity Catalog or its equivalent.
• Work with relational databases (Oracle, MySQL, SQL Server) and data modeling.
• Implement PySpark-based data processing; optimize queries and performance.
• Create Azure Data Factory pipelines for copy activity and pipelines in Databricks with CI/CD.
• Migrate data across Oracle to Azure Data Lake, Databricks, and Microsoft Fabric.

Skills

Data Engineering

CI/CD

DataOPS

ETL/ELT design

PySpark

Cloud data platforms

Agile

Problem solving

Communication

Team collaboration

Education

Bachelor's degree in Computer Science or related

Tools

Databricks

Delta Live Tables

Azure Data Factory

Unity Catalog

Azure DevOps

Oracle

SQL Server

Job description

• Designing and developing data pipelines from source to enduser
  

General Skills:

• Experience with Cloud data platforms, data management anddata exchange tools and technologies
  
• Experience with Commercial and opensource data and databasedevelopment and management and specializing in data storage setting andmanaging cloud Data As a Service (DaaS). application Database as a Service(DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms(both in the cloud and on-premise).
  
• Experience with Data pipeline and workflow development,orchestration, deployment, and automation and specializing in programming andpipelines to create and managing the dataflow and movement of data.
  
• Experience with Cloud data engineer must be familiar andexperienced with different programming languages and be able to integrate withmany different platforms to create data pipelines, automate tasks, and writescripts.
  
• Experience in DataOPS principle, best practice, andimplementation and Agile project development and deployment
  
• Experience in Continuous Integration/ContinuousDevelopment/Deployment (CI/CD) and Data provisioning Automation
  
• Experience with digital product, data analysis, dataexchange, data provisioning, and data security
  
• Extensive expert experience in designing/developing andimplementing data conversation and migration of VLD (Very large Data) of Onlineanalytical processing (OLAP) and online transaction processing (OLTP)environments to Cloud Software-as-a-service (SaaS), Platform-as-a-service(PaaS) and Infrastructure-as-a-service (IaaS) environments.
  
• Experience in design, development and implementation offact/dimension model, data mapping, data warehouse, data lake and datalakehouse for enterprise
  
• Experience managing Cloud Data services for projectdelivery, including storage, repositories, Data Lake, Data Lakehouse, keyvault, virtual machine, disk, etc.
  
• Experience with structured, semi-structured, unstructureddata collection, ingestion, provisioning and exchange technological developmentof enterprise data warehouse and data lake and data lakehouse solutions andoperational support.
  
• Experience with DataOPS performance monitoring and tuning
  
• Excellent analytical, problem-solving and decision-makingskills; verbal and written communication skills; presentation skills;interpersonal and negotiation skills
  
• A team player with a track record for meeting deadlines
  

Requirements

Experience and Skill Set Requirements:

Must Haves:

• Design,build, automate and optimize complex data ETL/ELT processes usingDatabricks, Delta Live Tables (DLT) and equivalent.
  
• Developmentof unified data platforms using Databricks and Unity catalog or itsequivalent.
  
• Relationaldatabases (Oracle, MySQL, SQL Server), data modelling (relational &dimensional), advanced SQL, query optimization, data replication,administration.
  
• BigData processing Frameworks (PySpark).
  
• Proficientin creating Azure Data Factory pipelines for copy activity and custom datapipelines in Databricks with CI CD (Azure DevOPS Git Integration) .
  
• Datamigration and data integration across different platforms Oracle to AzureData Lake, Oracle to Databricks, Oracle to Microsoft Fabric.
  

Skill Set Requirements:

Mandatory Skills and Experience (8+ years experience):

• Design,build, automate and optimize complex data ETL/ELT processes usingDatabricks, Delta Live Tables (DLT) and equivalent.
  
• Developmentof unified data platforms using Databricks and Unity catalog or itsequivalent.
  
• Relationaldatabases (Oracle, MySQL, SQL Server), data modelling (relational &dimensional), advanced SQL, query optimization, data replication,administration.
  
• BigData processing Frameworks (PySpark).
  
• Proficientin creating Azure Data Factory pipelines for copy activity and custom datapipelines in Databricks with CI CD (Azure DevOPS Git Integration) .
  
• Datamigration and data integration across different platforms Oracle to AzureData Lake, Oracle to Databricks, Oracle to Microsoft Fabric.
  

Desirable Skills and Experience:

• Datalake design and development with focus on extracting, transforming datacoming from various data sources and loading into Lakehouse in medallionarchitecture.
  
• Advancedskills and hands-on experience with Oracle databases, Azure SQL Server andAzure Data Factory.
  
• PL/SQLfor data extraction, transformation, and loading, initial setup and otherETL experience management and support, such as troubleshooting,performance tuning, failover and recovery.
  
• Scriptinglanguages (Python, Unix shell, Scala).