Senior ETL Developer - Databricks

CG Consulting Group; Toronto; Hybrid; CAD 80,000 - 120,000; Full time

Job summary

An established industry player is seeking a Senior ETL Developer with expertise in Databricks and data integration. This role involves designing and implementing ETL processes, transforming data for analytics, and ensuring compliance with enterprise architecture standards. The ideal candidate will have a strong background in financial services, excellent programming skills in Pyspark and SparkSQL, and a collaborative spirit. Join a dynamic team that values innovation and excellence in delivering strategic solutions to clients in the fintech sector. This position offers a hybrid work environment, allowing for a mix of home and office work, making it an exciting opportunity for professionals looking to make an impact.

Qualifications

• 5+ years in IT with complex projects, strong in data integration and analysis.
• Bachelor's degree in Computer Science required.

Responsibilities

• Design ETL processing and data integration using Databricks and Spark.
• Ingest and transform data for real-time processing.

Skills

Data Analysis

Communication Skills

Programming in Pyspark

Programming in SparkSQL

Knowledge of Data Integration Methodologies

Knowledge of Financial Industries

Education

Bachelor's Degree in Computer Science

Tools

Databricks

Spark

Hadoop

Hive

Azure Data Factory

Microsoft Azure

CI/CD Frameworks

Job description

Senior ETL Developer - Databricks

Client is a Toronto-based IT consulting boutique that strives for excellence in customer service and delivery in the financial services and fintech industry. They focus on technology advisory services, application development, cloud computing, integration solutions, and enterprise big data and analytics. With a team of highly knowledgeable business and technical experts, we provide strategic, mission-critical solutions to our clients.

Looking for a senior Databricks ETL Developer who has hands-on experience in gathering business requirements, designing Databricks solutions and data models, and building data pipelines in business analytics projects in the Greater Toronto Area.

Company location: Downtown Toronto,
Hybrid work environment: mix of home, office and client site
Perm / FTE role: Salary: Market rate + benefits and performance bonus

RESPONSIBILITIES

1. Understand business requirements for data and information especially in the financial services industry.
2. Design data integration or ETL processing using Databricks, Spark, Hadoop, and Hive for client engagements based on an approach that is holistic, scalable, pragmatic, and effective design of data processing.
3. Ingest data and transform data in Databricks for batch or near real-time processing using a variety of tools such as Pyspark, SparkSQL, Hive, Azure Data Factory, and Microsoft Azure.
4. Design and integrate pipelines with Continuous Integration/Delivery framework as needed.
5. Develop automated unit-testing.
6. Develop design and operational documentation for job schedules and data pipelines.
7. Produce estimates for data analysis and mapping, design, development, and testing.
8. Ensure compliance with business, data, and technical requirements.
9. Ensure that client’s enterprise architecture standards, policies, and procedures are followed.

REQUIREMENTS

1. Minimum 5 years experience in Information Technology, participating in complex projects.
2. Minimum Bachelors degree in Computer Science.
3. Strong communication skills.
4. Strong knowledge in data analysis, database development, big data, data lakehouse, data warehousing life cycle and data integration methodologies.
5. Strong knowledge of Databricks features and working experiences in developing data integration solutions using Databricks.
6. Strong programming skills in Pyspark and SparkSQL.
7. Working knowledge in Hadoop/Hive and one of Oracle or Microsoft SQL databases.
8. Knowledgeable in both relational and dimensional data modeling (with both Kimball and Inmon approaches).
9. Experience in .NET development and UNIX shell scripting.
10. Knowledge in data management and Continuous Integration and Delivery (CI/CD) principles.
11. Good overall business knowledge in financial industries (i.e. one or more in retail banking, commercial banking, capital markets, wealth management, insurance, pension fund, and fintech).
12. Knowledgeable of Azure computing concepts and services.
13. Ability to work independently and excel in a team environment.