Senior Data Engineer Lead- ScotiaTech

Scotiabank; Bogotá; Presencial; COP 120.000.000 - 260.000.000; Jornada completa

Descripción de la vacante

Scotiabank seeks a Senior Data Engineer in Bogotá to design, build and operate scalable data products and pipelines on Azure Databricks. The role leads a small team, owning end-to-end design and implementation, while ensuring governance and regulatory compliance across International Banking markets.

You will mentor engineers, establish engineering standards, and collaborate with data scientists, product owners, and platform teams to deliver secure, cost-efficient lakehouse solutions at scale.

Formación

• Bachelor’s degree in Computer Science, Data Engineering, IT, or related field.
• Master’s degree is an asset.

Responsabilidades

• Design, build, test, and optimize data pipelines and lakehouse solutions on Azure Databricks.
• Lead, mentor, and develop a team of Data Engineers; oversee design reviews and code quality.

Conocimientos

Azure Databricks

Python

SQL

Spark

Data Modeling

CI/CD

Azure DevOps

Security & Compliance

Educación

Bachelor’s degree in Computer Science or related field

Master’s degree (asset)

Herramientas

Azure Data Factory

Azure Synapse

Unity Catalog

Terraform

GitHub Actions

Descripción del empleo

Purpose

The Senior Data Engineer (Level 8) designs, builds, and operates scalable data products and pipelines on Microsoft Azure and Azure Databricks, with a strategic focus on International Banking initiatives. This role is pivotal in driving data-driven decision-making by delivering robust, efficient, and secure lakehouse-based data solutions. Reporting to the Director, Data Engineering, the incumbent is the hands-on technical authority for the solutions in scope while leading a small team of Data Engineers: owning design and implementation end-to-end, setting engineering standards, developing and mentoring the team, and ensuring data solutions comply with enterprise architecture, security, and regulatory requirements across International Banking markets.

Accountabilities

• Design, build, test, and optimize data pipelines and lakehouse solutions on Azure and Azure Databricks (Delta Lake, medallion architecture) to support scalable data products for International Banking.
• Act as the hands-on technical lead for assigned initiatives: own the end-to-end solution design, produce the technical documentation, and deliver production-grade code.
• Lead, mentor, and develop a team of Data Engineers, including work assignment, coaching, performance feedback, and participation in hiring; lead code and design reviews and promote a culture of engineering excellence, collaboration, and continuous improvement.
• Provide effort estimates, technical risks, and dependencies to support delivery planning, allocate work across the team, and keep the Director informed of progress and blockers.
• Collaborate with cross-functional partners (Data Scientists, Product Owners, Architecture, Platform Engineering, and Software Engineering) to translate business requirements into data products aligned with international business goals.
• Apply and help enforce best practices for data governance, security, and compliance within Azure environments, including Unity Catalog, role-based access control, data lineage, encryption, and key management.
• Monitor and maintain data pipelines to ensure high availability, data quality, cost efficiency (FinOps), and efficient processing across international markets.
• Champion the adoption of modern data engineering tools and methodologies, including ETL/ELT with Azure Data Factory and Databricks Workflows, dimensional and Data Vault modelling, DevOps/DataOps practices, and orchestration at scale.
• Troubleshoot and resolve complex data pipeline and platform incidents, performing root-cause analysis and implementing permanent fixes to meet agreed service levels.
• Support the migration and rationalization of legacy workloads (including existing Google Cloud Platform estates) onto the Azure Databricks target platform, minimizing business and regulatory risk.
• Evaluate and pilot new Azure and Databricks services and third-party tools, recommending adoption where they enhance data capabilities and support international expansion.

Dimensions

Colombia, Canada

Business Unit: IBTT

Player-coach role within a multi-country data engineering team supporting International Banking, combining hands-on delivery with the leadership of a small team of Data Engineers. Accountable for the technical delivery and operational health of the data pipelines and data products in scope, typically working across one or more concurrent initiatives in global markets, and for the efficient consumption of Azure Databricks resources associated with those workloads.

Education / Experience / Other Information

Education

• Bachelor’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• A Master’s degree is considered an asset.

Experience

• 5+ years of progressive hands-on experience in data engineering, including at least 3 years delivering production data solutions on Microsoft Azure and/or Azure Databricks.
• Demonstrated experience as a technical lead on complex data initiatives, including leading design and code reviews and directly supervising, coaching, and developing data engineers.
• Expertise in designing and implementing data pipelines using Azure Databricks (Spark, Delta Lake, Unity Catalog, Workflows), Azure Data Factory, Azure Data Lake Storage Gen2, Event Hubs, and Azure Synapse or equivalent serving layers.
• Advanced proficiency in Python, PySpark, and SQL (Scala or Java an asset) for large-scale batch and streaming data processing, including performance tuning and cost optimization of Spark workloads.
• Hands-on experience with data modelling, schema design, lakehouse/medallion patterns, and data warehousing concepts.
• Strong working knowledge of CI/CD pipelines (Azure DevOps or GitHub Actions), version control (Git), Infrastructure as Code (Terraform or Bicep), automated testing, and Databricks Asset Bundles or equivalent deployment tooling.
• Knowledge of data governance, security, and compliance standards in cloud environments (e.g., GDPR, local data residency and privacy regulations), including Microsoft Entra ID, RBAC, and cloud key management.
• Experience migrating workloads from another public cloud or on-premises platform (e.g., GCP BigQuery/Dataflow to Azure Databricks) is a strong asset; Azure or Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Azure DP-203) are an asset.
• Excellent problem-solving, influencing, and communication skills, with the ability to present complex technical concepts to non-technical stakeholders; Spanish language ability is an asset given the International Banking footprint.

Soft Skills:

• Strong leadership, stakeholder communication, and problem-solving abilities. Candidates must demonstrate adaptability and a growth mindset.
• Results-oriented mindset with a strong focus on execution, quality, and delivery.
• Strong communicationskills (written & verbal), interacting with Product, Design and Back-end Teams.
• Strong sense of ownership, accountability, and commitment to achieving business outcomes.
• Proactive approach to identifying risks, removing obstacles, and driving continuous improvement.
• Ability to manage multiple priorities while maintaining a sense of urgency and attention to detail.
• Ability to work effectively in collaborative, distributed, and multicultural teams.
• Strong verbal and written English communication skills (B2+ or higher).

Working Conditions

Work in a standard office-based environment , with standard hours .