Specialized IT Consultant - Senior

LanceSoft Inc; Toronto; On-site; CAD 80,000 - 120,000; Full time

Job summary

An established industry player is seeking a Senior ETL Developer/Specialized IT Consultant to join their dynamic team. In this role, you will leverage your extensive experience in developing and managing ETL pipelines, utilizing cutting-edge tools like Databricks and Delta Lake. Your expertise will be crucial in ensuring data quality and optimizing data workflows, while also collaborating with cross-functional teams to communicate complex concepts effectively. This position offers a unique opportunity to work on innovative projects that drive data-driven decision-making, making a significant impact on the organization. If you are passionate about data engineering and eager to contribute to transformative initiatives, this role is perfect for you.

Qualifications

• 7+ years of experience with ETL tools and data management.
• Strong skills in Python and PySpark for data manipulation.
• Expertise in Delta Lake and Databricks for data optimization.

Responsibilities

• Develop and manage ETL pipelines and workflows in Databricks.
• Create design documentation and troubleshoot ETL processes.
• Collaborate with teams and provide technical guidance.

Skills

ETL Development

Delta Lake

Databricks

Python

PySpark

SQL

Data Governance

Change Data Capture (CDC)

Medallion Architecture

Data Quality Assurance

Tools

Microsoft SSIS

Azure Data Factory

Unity Catalog

GoldenGate

SQL Server

Oracle

Job description

Title: ETL Developer/Specialized IT Consultant - Senior

Location: 5775 YONGE STREET FLOOR 4 TORONTO ONTARIO M2M 4J1

Duration: 248 days

Skills

Experience and Skill Set Requirements

Technical Skills - 70 points

1. Experience in developing and managing ETL pipelines, jobs, and workflows in Databricks.
2. Deep understanding of Delta Lake for building data lakes and managing ACID transactions, schema evolution, and data versioning.
3. Experience automating ETL pipelines using Delta Live Tables, including handling Change Data Capture (CDC) for incremental data loads.
4. Proficient in structuring data pipelines with the Medallion Architecture to scale data pipelines and ensure data quality.
5. Hands-on experience developing streaming tables in Databricks using Structured Streaming and readStream to handle real-time data.
6. Expertise in integrating CDC tools like GoldenGate or Debezium for processing incremental updates and managing real-time data ingestion.
7. Experience using Unity Catalog to manage data governance, access control, and ensure compliance.
8. Skilled in managing clusters, jobs, autoscaling, monitoring, and performance optimization in Databricks environments.
9. Knowledge of using Databricks Autoloader for efficient batch and real-time data ingestion.
10. Experience with data governance best practices, including implementing security policies, access control, and auditing with Unity Catalog.
11. Proficient in creating and managing Databricks Workflows to orchestrate job dependencies and schedule tasks.
12. Strong knowledge of Python, PySpark, and SQL for data manipulation and transformation.
13. Experience integrating Databricks with cloud storage solutions such as Azure Blob Storage, AWS S3, or Google Cloud Storage.
14. Familiarity with external orchestration tools like Azure Data Factory.
15. Implementing logical and physical data models.
16. Knowledge of FHIR is an asset.

Design Documentation and Analysis Skills - 20 points

1. Demonstrated experience in creating design documentation such as:
2. Schema definitions.
3. Error handling and logging.
4. ETL Process Documentation.
5. Job Scheduling and Dependency Management.
6. Data Quality and Validation Checks.
7. Performance Optimization and Scalability Plans.
8. Troubleshooting Guides.
9. Data Lineage.
10. Security and Access Control Policies applied within ETL.
11. Experience in Fit-Gap analysis, system use case reviews, requirements reviews, coding exercises, and reviews.
12. Participate in defect fixing, testing support, and development activities for ETL.
13. Analyze and document solution complexity and interdependencies including providing support for data validation.
14. Strong analytical skills for troubleshooting, problem-solving, and ensuring data quality.

Communication and Leadership Skills - 10 points

1. Ability to collaborate effectively with cross-functional teams and communicate complex technical concepts to non-technical stakeholders.
2. Strong problem-solving skills and experience working in an Agile or Scrum environment.
3. Ability to provide technical guidance and support to other team members on Databricks best practices.
4. Must have previous work experience in conducting Knowledge Transfer sessions, ensuring the resources will receive the required knowledge to support the system.
5. Must develop documentation and materials as part of a review and knowledge transfer to other members.

Must Have Skills

1. 7+ years using ETL tools such as Microsoft SSIS, stored procedures, T-SQL.
2. 2+ years with Delta Lake, Databricks, and Azure Databricks pipelines.
3. Strong knowledge of Delta Lake for data management and optimization.
4. Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
5. 2+ years Python and PySpark.
6. Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments.
7. Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data.
8. Experience with SQL Server and Oracle.