Lead Data Engineer - API-Driven on Azure Databricks & DBT

Encora Inc.; Medellín; Remote; COP 279.520.000 - 403.752.000; Vollzeit

Zusammenfassung

Coforge seeks a Lead Data Engineer to drive end-to-end data product delivery on a modern cloud platform. You will own architecture, guide a team of data and ML engineers, and collaborate with Product, Engineering, and Business stakeholders.

Expertise in Python, Azure Databricks, DBT, SQL, Terraform, and CI/CD, plus strong API design and data modeling, is required to build scalable, low-latency solutions with observable data pipelines and robust QA.

Qualifikationen

• 8+ years of experience in Data Engineering, Software Engineering, or related technical roles.
• At least 3 years of experience leading engineering teams, including senior, mid-level, and junior professionals.
• Strong backend development experience with proven delivery of production-grade REST APIs and backend services.
• Expert-level programming skills in Python, including strong object-oriented programming (OOP) principles.
• Extensive experience with PyTest, unit testing, automated testing frameworks, and software quality practices.
• Strong hands-on experience with Azure Databricks and DBT.
• Advanced SQL skills and experience building scalable data solutions.
• Deep understanding of data modeling, dimensional modeling, and consumption-oriented data architectures.
• Extensive experience with Terraform and Infrastructure as Code (IaC) deployments.
• Experience designing and delivering low-latency data products, serving layers, or API-driven data solutions.
• Strong knowledge of CI/CD pipelines, Git-based version control, and modern software engineering practices.
• Proven ability to lead technical delivery while remaining hands-on with architecture, development, and code reviews.
• Excellent communication and stakeholder management skills, with the ability to work effectively across technical and business teams.

Aufgaben

• Lead the RTS delivery squad and own the end-to-end delivery of strategic data products within a modern cloud data platform.
• Design and oversee the architecture of low-latency, API-driven data solutions, from backend data models through serving layers and API consumption.
• Build and maintain scalable backend services and REST APIs using Python and object-oriented programming principles.
• Design, develop, and optimize data pipelines using Azure Databricks, DBT, SQL, and cloud-native technologies.
• Own the architecture and implementation of curated data layers, ensuring high availability, scalability, and performance.
• Evaluate, select, and implement technologies for low-latency data serving and API consumption.
• Lead infrastructure deployment and management using Terraform and Infrastructure as Code (IaC) practices.
• Mentor and develop a team of data and machine learning engineers, providing technical guidance, coaching, and career development support.
• Conduct code reviews and establish engineering standards, development best practices, and architectural guidelines.
• Collaborate with Product, Engineering, Business, and Project stakeholders to define technical solutions and delivery roadmaps.
• Ensure data quality, observability, monitoring, alerting, and SLA compliance across all data products and services.
• Drive CI/CD adoption and oversee release management processes for pipelines, services, and platform components.
• Act as the primary technical escalation point for troubleshooting complex production issues and ensuring long-term platform stability.
• Maintain comprehensive technical documentation, architectural decisions, and implementation standards.

Kenntnisse

Python

REST APIs

Azure Databricks

DBT

SQL

Terraform

Data Engineering

Data Modeling

CI/CD

Team Leadership

Tools

Python Tests (PyTest)

Jobbeschreibung

Coforge seeks a Lead Data Engineer to drive end-to-end data product delivery on a modern cloud platform. You will own architecture, guide a team of data and ML engineers, and collaborate with Product, Engineering, and Business stakeholders.

Expertise in Python, Azure Databricks, DBT, SQL, Terraform, and CI/CD, plus strong API design and data modeling, is required to build scalable, low-latency solutions with observable data pipelines and robust QA.