DataOps Engineer Specialist

Scotiabank; Bogotá; Presencial; COP 90.000.000 - 140.000.000; Jornada completa

Descripción de la vacante

Scotiabank in Bogota seeks a DataOps Engineer Specialist to enable data integration and DataOps across International Banking. You will work hands-on with architecture and senior engineers to migrate legacy on-prem and GCP data to Azure Databricks, ensuring trusted, well‑understood data for migration in multiple markets.

The role requires 3+ years in DataOps or data engineering, strong PySpark/Python, SQL expertise, and experience with Git, CI/CD, and Agile teams.

Formación

• Bachelor’s degree in Computer Science, Data Engineering, IT or related field.
• 3+ years of DataOps, data integration, or data engineering experience.
• Experience mapping data and supporting data migration or cloud modernization.
• Strong SQL skills and large-scale data warehouse experience.
• Familiarity with DataOps practices (Git, CI/CD, testing, monitoring).
• Ability to work with SMEs and cross-functional teams in Agile delivery.

Responsabilidades

• Integrate data across sources including on‑prem to Azure Databricks and validate end‑to‑end flows.
• Deploy approved ingestion, transformation and integration patterns with architecture team.
• Support DataOps: Git, CI/CD, automated tests, monitoring to keep pipelines reliable.
• Discover and map legacy and GCP data to Delta Lake model with lineage documentation.
• Ensure data quality and parity across Bronze→Silver→Gold medallion layers.
• Collaborate with SMEs and catalog data in Unity Catalog for discoverability.
• Build simple reports/dashboards (Power BI) showing data availability and migration progress.

Conocimientos

Data integration

Apache Spark (PySpark)

Python

SQL

DataOps practices

Git

CI/CD

Agile

Cross-functional collaboration

Educación

Bachelor's degree in Computer Science / Data Engineering / IT / Software Engineering

Master's degree is a plus

Herramientas

Azure Databricks

Delta Lake

Unity Catalog

Power BI

GCP / AWS

Descripción del empleo

Purpose

The DataOps Engineer Specialist enables data integration and DataOps for International Banking, supporting the reliable flow of data across systems and into Harbour (Azure Databricks). Working hands‑on with architecture and senior engineers, this role integrates data, deploys approved data patterns, and helps discover, map, and articulate legacy on‑premises and GCP data so it can be migrated with confidence. Reporting to the Senior Data Engineer, the incumbent partners with SMEs and the engineering team to make data trusted, well‑understood, and ready for use across International Banking markets.

Accountabilities

• Data Integration: Integrate data across sources—including legacy on‑premises into Azure Databricks—and validate end‑to‑end data flow so data lands complete and usable.
• Pattern Deployment & Architecture: Deploy and reuse the team’s approved ingestion, transformation, and integration patterns, partnering with architecture and senior engineers to follow the target design and standards.
• DataOps & Automation: Support DataOps practices (Git, CI/CD, automated testing, monitoring) to keep pipelines reliable, available, and performant.
• Discovery & Mapping: Explore legacy and GCP data, produce source‑to‑target mappings to the Delta Lake model, and document lineage and dependencies to support migration.
• Data Quality & Reconciliation: Check data quality and reconcile source vs. target (parity), cleansing and preparing data across the medallion layers (Bronze→Silver→Gold).
• SME Collaboration & Articulation: Work with SMEs to understand and clearly articulate what data means, and support cataloguing in Unity Catalog so data is discoverable and trusted.
• Reporting & Visibility: Build simple reports/dashboards (e.g., Power BI) on data availability and migration progress—by country, source, and owner.

Reporting Relationships

Primary Manager:

Direct Reports:

Shared Reports

NA

Dimensions

NA

Education / Experience / Other Information (include only those that are specific to the role)

Education

Bachelor’s degree in Computer Science, Data Engineering, Information Technology, Software Engineering, or a related field.

A Master’s degree is considered a plus.

Relevant cloud or data engineering certifications (Databricks, GCP, Azure, AWS) are considered a plus.

Experience

3+ years of experience in DataOps, data integration, or data engineering support.

Experience integrating and mapping data, and supporting data migration or cloud modernization (on‑prem or GCP to Azure Databricks is an asset).

Familiarity with DataOps practices (Git, CI/CD, automated testing, monitoring).

Experience working with SMEs and cross‑functional teams in an Agile environment.

Strong proficiency in Apache Spark (PySpark) and Python for large‑scale data processing.

Knowledge of data migration, cloud modernization, and platform transformation initiatives is highly desirable.

Strong SQL skills and experience working with large‑scale analytical databases and data warehouses.

Experience with data modeling, schema design, and data warehousing concepts.

Understanding data governance, security, monitoring, and observability practices within cloud environments.

Experience working in Agile delivery environments and collaborating with cross‑functional teams.

Understanding of data integration, ETL/ELT, and lakehouse/medallion patterns; Azure Databricks, Delta Lake, and Unity Catalog exposure is an asset.

Preferred Qualifications

Databricks certifications (Data Engineer Associate/Professional) are highly desirable; Azure DP-203 is an asset.

Experience with Lakehouse architecture and Delta Lake implementation.

Familiarity with infrastructure-as-code tools such as Terraform.

Exposure to data quality, data observability, and monitoring frameworks.

Experience supporting cloud migration initiatives from on‑premises or legacy platforms.

Working Conditions

Work in a standard office-based environment; non‑standard hours are a common occurrence