Senior Data Engineer Lead

ScotiaTech; Colombia; Presencial; COP 180.000.000 - 320.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Multicultural environment

Well-being benefits

Competitive wages

Descripción de la vacante

Scotiatech is seeking a Senior Data Engineer Lead to design, build, and optimize data pipelines on Azure Databricks for its International Banking initiatives. You will lead a team, deliver production-grade code, and ensure data governance and security across cloud environments.

The role requires strong Python, PySpark, and SQL skills, plus experience in leading complex data projects and collaborating with cross-functional teams. Fluency in English is required; Spanish is a plus.

Formación

• Bachelor’s degree in Computer Science, Data Engineering, IT or related field.
• Master’s degree is asset.
• 5+ years in data engineering with 3+ years on Azure/Azure Databricks.
• Strong lead experience: design reviews, coaching, and supervision.
• Proficient in Python, PySpark, SQL; Scala/Java asset.

Responsabilidades

• Design, build, test, and optimize data pipelines and lakehouse on Azure Databricks.
• Lead end-to-end solution design; produce docs and production-grade code.
• Mentor data engineers; oversee hiring, code reviews, and engineering excellence.
• Provide estimates, risks, and dependencies; report blockers to leadership.
• Collaborate with Data Scientists, PO, Architecture, Platform, and Software Eng.

Conocimientos

Python

PySpark

SQL

Data governance

Cloud data engineering

Educación

Bachelor’s degree in Computer Science or related field

Master’s degree asset

Herramientas

Azure Databricks

Azure

Terraform

GitHub Actions

Azure DevOps

Databricks Asset Bundles

Descripción del empleo

Scotiatech is a world-class engineering group! We are seeking to modernize bank technology by delivering a trustworthy, unique, and top-quality service.

We want you to be part of a Team that is always in personal and professional growth, for this reason we are looking for our new Senior Data Engineer Lead.

What will you do?

• Design, build, test, and optimize data pipelines and lakehouse solutions on Azure and Azure Databricks (Delta Lake, medallion architecture) to support scalable data products for International Banking.
• Act as the hands-on technical lead for assigned initiatives: own the end-to-end solution design, produce the technical documentation, and deliver production-grade code.
• Lead, mentor, and develop a team of Data Engineers, including work assignment, coaching, performance feedback, and participation in hiring; lead code and design reviews and promote a culture of engineering excellence, collaboration, and continuous improvement.
• Provide effort estimates, technical risks, and dependencies to support delivery planning, allocate work across the team, and keep the Director informed of progress and blockers.
• Collaborate with cross-functional partners (Data Scientists, Product Owners, Architecture, Platform Engineering, and Software Engineering) to translate business requirements into data products aligned with international business goals.
• Apply and help enforce best practices for data governance, security, and compliance within Azure environments, including Unity Catalog, role-based access control, data lineage, encryption, and key management.
• Monitor and maintain data pipelines to ensure high availability, data quality, cost efficiency (FinOps), and efficient processing across international markets.
• Champion the adoption of modern data engineering tools and methodologies, including ETL/ELT with Azure Data Factory and Databricks Workflows, dimensional and Data Vault modelling, DevOps/DataOps practices, and orchestration at scale.
• Troubleshoot and resolve complex data pipeline and platform incidents, performing root-cause analysis and implementing permanent fixes to meet agreed service levels.
• Support the migration and rationalization of legacy workloads (including existing Google Cloud Platform estates) onto the Azure Databricks target platform, minimizing business and regulatory risk.
• Evaluate and pilot new Azure and Databricks services and third-party tools, recommending adoption where they enhance data capabilities and support international expansion.

What do we offer you?

• Growth opportunities
• Diverse, dynamic and multicultural environment
• Benefits for your financial and emotional well-being
• competitive wages

What do we expect from you?

Education

• Bachelor’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• A Master’s degree is considered an asset.

Experience

• Strong verbal and written English communication skills (B2+ or higher).
• 5+ years of progressive hands-on experience in data engineering, including at least 3 years delivering production data solutions on Microsoft Azure and/or Azure Databricks.
• Demonstrated experience as a technical lead on complex data initiatives, including leading design and code reviews and directly supervising, coaching, and developing data engineers.
• Advanced proficiency in Python, PySpark, and SQL (Scala or Java an asset) for large-scale batch and streaming data processing, including performance tuning and cost optimization of Spark workloads.
• Strong working knowledge of CI/CD pipelines (Azure DevOps or GitHub Actions), version control (Git), Infrastructure as Code (Terraform or Bicep), automated testing, and Databricks Asset Bundles or equivalent deployment tooling.
• Knowledge of data governance, security, and compliance standards in cloud environments (e.g., GDPR, local data residency and privacy regulations), including Microsoft Entra ID, RBAC, and cloud key management.
• Experience migrating workloads from another public cloud or on-premises platform (e.g., GCP BigQuery/Dataflow to Azure Databricks) is a strong asset; Azure or Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Azure DP-203) are an asset.
• Excellent problem-solving, influencing, and communication skills, with the ability to present complex technical concepts to non-technical stakeholders; Spanish language ability is an asset given the International Banking footprint.