Enable job alerts via email!

Specialized IT Consultant

Integrated Resources, Inc.

Toronto

On-site

CAD 90,000 - 120,000

Full time

2 days ago
Be an early applicant

Boost your interview chances

Create a job specific, tailored resume for higher success rate.

Job summary

A leading company is seeking a Specialized IT Consultant for a one-year contract in Toronto. The role focuses on data architecture and ETL processes, requiring expertise in Azure Databricks and Delta Lake. The successful candidate will collaborate with business teams to enhance data management and analytics. This position involves significant hands-on work with data pipelines and requires strong knowledge of ETL tools and methodologies.

Qualifications

  • 7+ years using ETL tools like Microsoft SSIS, T-SQL.
  • 2+ years with Delta Lake and Azure Databricks pipelines.
  • Strong knowledge of Medallion Architecture.

Responsibilities

  • Design, develop, and optimize ETL processes in Databricks.
  • Analyze requirements and recommend changes to data models.
  • Perform data profiling and ensure data quality.

Skills

ETL tools
Delta Lake
Python
PySpark
SQL

Tools

Azure Databricks
Delta Lake
SQL Server
Oracle

Job description

Job Title: Specialized IT Consultant
Location: Toronto, ON (3 days onsite/week)
Duration: 1 year Contract

Job Description:
Position Summary:
This role will focus on data architecture, data warehousing, data lakes, and analytics. The individual will be designing, developing, maintaining, and optimizing ETL (Extract, Transform, Load) processes in Databricks for data warehousing, data lakes, and analytics. The individual will work closely with data architects and business teams to ensure the efficient transformation and movement of data to meet business needs, including handling Change Data Capture (CDC) and streaming data.

Roles & Responsibilities:

  • Review business requirements, familiarize with and understand business rules and transactional data model
  • Define conceptual, logical model and physical model mapping from data source to curated model and data mart.
  • Analyze requirements and recommend changes to the physical model.
  • Develop scripts for the physical model, create database and/or delta lake file structure.
  • Access Oracle DB environments, set necessary tools for developing solution.
  • Implement data design methodologies, historical and dimensional models
  • Perform data profiling, assess data accuracy, design and document data quality and master data management rules
  • Functionality Review, Data Load review, Performance Review, Data Consistency checks.
  • Help troubleshooting data mart design issues
  • Review performance of ETL with developers and suggest improvements
  • Participate in end-to-end integrated testing for Full Load and Incremental Load and advise on issues

Tools used are:
  • Azure Databricks, Delta Lake, Delta Live Tables, and Spark to process structured and unstructured data.
  • Azure Databricks/PySpark (good Python/PySpark knowledge required) to build transformations of raw data into curated zone in the data lake.
  • Azure Databricks/PySpark/SQL (good SQL knowledge required) to develop and/or troubleshoot transformations of curated data into FHIR.

Data design:
  • Understand the requirements. Recommend changes to models to support ETL design.
  • Define primary keys, indexing strategies, and relationships that enhance data integrity and performance across layers.
  • Define the initial schemas for each data layer
  • Assist with data modelling and updates of source-to-target mapping documentation
  • Document and implement schema validation rules to ensure incoming data conforms to expected formats and standards
  • Design data quality checks within the pipeline to catch inconsistencies, missing values, or errors early in the process.
  • Proactively communicate with business and IT experts on any changes required to conceptual, logical and physical models, communicate and review timelines, dependencies, and risks.
  • Development of ETL strategy and solution for different sets of data modules
  • Understand the Tables and Relationships in the data model.
  • Create low level design documents and test cases for ETL development.
  • Implement error-catching, logging, retry mechanisms, and handling data anomalies.
  • Create the workflows and pipeline design
  • Development and testing of data pipelines with Incremental and Full Load.
  • Develop high quality ETL mappings/scripts/notebooks
  • Develop and maintain pipeline from Oracle data source to Azure Delta Lakes and FHIR
  • Perform unit testing
  • Ensure performance monitoring and improvement
  • Performance review, data consistency checks
  • Troubleshoot performance issues, ETL issues, log activity for each pipeline and transformation.
  • Review and optimize overall ETL performance.
  • End-to-end integrated testing for Full Load and Incremental Load
  • Plan for Go Live, Production Deployment.
  • Create production deployment steps.
  • Configure parameters, scripts for go live. Test and review the instructions.
  • Create release documents and help build and deploy code across servers.
  • Go Live Support and Review after Go Live.
  • Review existing ETL process, tools and provide recommendation on improving performance and reduce ETL timelines.
  • Review infrastructure and remediate issues for overall process improvement
  • Knowledge Transfer to Ministry staff, development of documentation on the work completed.
  • Document work and share the ETL end-to-end design, troubleshooting steps, configuration and scripts review.
  • Transfer documents, scripts and review of documents to Ministry.

Must Have Skills:
  • 7+ years using ETL tools such as Microsoft SSIS, stored procedures, T-SQL
  • 2+ Delta Lake, Databricks and Azure Databricks pipelines
  • Strong knowledge of Delta Lake for data management and optimization.
  • Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
  • 2+ years Python and PySpark
  • Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments.
  • Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data.
  • SQL Server, Oracle
Get your free, confidential resume review.
or drag and drop a PDF, DOC, DOCX, ODT, or PAGES file up to 5MB.

Similar jobs

Solutions Consultant - AI Specialist

Thomas Reuters

Toronto

Remote

CAD 80,000 - 100,000

3 days ago
Be an early applicant

Freelance Political Science Expert - AI Tutor (Talent pool)

Mindrift

Toronto

Remote

CAD 80,000 - 100,000

2 days ago
Be an early applicant

Solutions Consultant - AI Specialist

Thomas Reuters

Toronto

Remote

CAD 70,000 - 110,000

14 days ago

Freelance Material Science Expert - AI Tutor (Talent pool)

Mindrift

Toronto

Remote

CAD 60,000 - 100,000

15 days ago

IT Consultant

vTech Solution

Ontario

Remote

CAD 80,000 - 120,000

10 days ago

Senior Specialized IT Consultant

Akkodis group

Toronto

Hybrid

CAD 100,000 - 130,000

3 days ago
Be an early applicant

RQ09148 - Specialized IT Consultant - Senior

Rubicon Path

Toronto

On-site

CAD 60,000 - 100,000

12 days ago

Senior Specialized IT Consultant

CYNET SYSTEMS

Toronto

On-site

CAD 80,000 - 110,000

10 days ago

Freelance Medicine Expert - AI Tutor

Mindrift

Toronto

Remote

CAD 80,000 - 100,000

30+ days ago