Senior ETL Developer - Databricks

Senior ETL Developer - Databricks

Client is a Toronto-based IT consulting boutique that strives for excellence in customer service and delivery in the financial services and fintech industry. They focus on technology advisory services, application development, cloud computing, integration solutions, and enterprise big data and analytics. With a team of highly knowledgeable business and technical experts, we provide strategic, mission-critical solutions to our clients.

Looking for a senior Databricks ETL Developer who has hands-on experience in gathering business requirements, designing Databricks solutions and data models, and building data pipelines in business analytics projects in the Greater Toronto Area.

Company location: Downtown Toronto,
Hybrid work environment: mix of home, office and client site
Perm / FTE role: Salary: Market rate + benefits and performance bonus

RESPONSIBILITIES

1. Understand business requirements for data and information especially in the financial services industry.
2. Design data integration or ETL processing using Databricks, Spark, Hadoop, and Hive for client engagements based on an approach that is holistic, scalable, pragmatic, and effective design of data processing.
3. Ingest data and transform data in Databricks for batch or near real-time processing using a variety of tools such as Pyspark, SparkSQL, Hive, Azure Data Factory, and Microsoft Azure.
4. Design and integrate pipelines with Continuous Integration/Delivery framework as needed.
5. Develop automated unit-testing.
6. Develop design and operational documentation for job schedules and data pipelines.
7. Produce estimates for data analysis and mapping, design, development, and testing.
8. Ensure compliance with business, data, and technical requirements.
9. Ensure that client’s enterprise architecture standards, policies, and procedures are followed.

REQUIREMENTS

1. Minimum 5 years experience in Information Technology, participating in complex projects.
2. Minimum Bachelors degree in Computer Science.
3. Strong communication skills.
4. Strong knowledge in data analysis, database development, big data, data lakehouse, data warehousing life cycle and data integration methodologies.
5. Strong knowledge of Databricks features and working experiences in developing data integration solutions using Databricks.
6. Strong programming skills in Pyspark and SparkSQL.
7. Working knowledge in Hadoop/Hive and one of Oracle or Microsoft SQL databases.
8. Knowledgeable in both relational and dimensional data modeling (with both Kimball and Inmon approaches).
9. Experience in .NET development and UNIX shell scripting.
10. Knowledge in data management and Continuous Integration and Delivery (CI/CD) principles.
11. Good overall business knowledge in financial industries (i.e. one or more in retail banking, commercial banking, capital markets, wealth management, insurance, pension fund, and fintech).
12. Knowledgeable of Azure computing concepts and services.
13. Ability to work independently and excel in a team environment.