Senior Data Engineer Lead

Scotiatech Limited; Bogotá; On-site; COP 284,765,000 - 442,968,000; Full time

Job summary

Scociatech Limited in Bogotá seeks a Senior Data Engineer to design, build, and maintain scalable data products and pipelines across AWS, Azure, and GCP. You will lead a small team, drive data-driven decision making, and ensure robust, secure data infrastructure.

The role emphasizes governance, security, and performance, with collaboration across Data Scientists, Product Managers, and Software Engineers to deliver data solutions that advance business goals.

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, IT, or related field.
• Master’s degree is an asset.

Responsibilities

• Design, build, and optimize data pipelines and architectures on cloud platforms to support data products, ensuring scalability, reliability, and performance.
• Lead a team of up to three Data Engineers, providing mentorship, technical direction, and performance feedback.
• Collaborate with cross-functional teams to understand requirements and deliver data products that align with business goals.
• Implement best practices for data governance, security, and compliance within cloud environments.
• Monitor and maintain data infrastructure, ensuring high availability, data quality, and efficient processing.
• Drive the adoption of modern data engineering tools and methodologies, including ETL/ELT processes, data modeling, and orchestration.
• Troubleshoot and resolve complex data pipeline issues, ensuring minimal downtime and optimal performance.
• Contributes to the evaluation and integration of new cloud services and third-party tools to enhance data capabilities.

Skills

Python

Java

Scala

SQL

Data modeling

CI/CD

Git

Terraform

Education

Bachelor’s degree in Computer Science or related field

Master’s degree preferred

Tools

Azure

AWS

GCP

Databricks

BigQuery

Dataflow

Dataproc

Pub/Sub

Cloud Composer

Git

Terraform

Job description

The Senior Data Engineer will lead the design, development, and maintenance of scalable data products and pipelines across multiple cloud platforms, including AWS, Microsoft Azure, and Google Cloud Platform (GCP). This role is critical in driving data-driven decision-making by building robust, efficient, and secure data infrastructure in multi-cloud environments. The Senior Data Engineer will oversee a team of up to three Data Engineers, providing technical guidance and ensuring high-quality delivery of data solutions that support business objectives.

Accountabilities

• Design, build, and optimize data pipelines and architectures on cloud platforms to support data products, ensuring scalability, reliability, and performance.
• Lead a team of up to three Data Engineers, providing mentorship, technical direction, and performance feedback.
• Collaborate with cross-functional teams (e.g., Data Scientists, Product Managers, and Software Engineers) to understand requirements and deliver data products that align with business goals.
• Implement best practices for data governance, security, and compliance within cloud environments.
• Monitor and maintain data infrastructure, ensuring high availability, data quality, and efficient processing.
• Drive the adoption of modern data engineering tools and methodologies, including ETL/ELT processes, data modeling, and orchestration.
• Troubleshoot and resolve complex data pipeline issues, ensuring minimal downtime and optimal performance.
• Contributes to the evaluation and integration of new cloud services and third-party tools to enhance data capabilities.

Education / Experience / Other Information

Education

• Bachelor’s degree in computer science, Data Engineering, Information Technology, or a related field.
• A master’s degree is considered an asset.

Experience

• 6+ years of experience in data engineering, with at least 3 years focused on building data solutions across cloud platforms, including Microsoft Azure, AWS, and Google Cloud Platform (GCP). Specific hands-on experience with Microsoft Azure and Databricks is highly desirable.
• Proven expertise in designing and implementing data pipelines using GCP tools such as BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Composer.
• 2+ years of experience leading or mentoring a team of engineers, with a track record of successful project delivery.
• Strong proficiency in programming languages such as Python, Java, or Scala, and SQL for data processing.
• Hands-on experience with data modeling, schema design, and data warehousing concepts.
• Familiarity with CI/CD pipelines, version control (e.g., Git), and Infrastructure as Code (e.g., Terraform) is preferred.
• Excellent problem-solving skills and the ability to communicate complex technical concepts to non-technical stakeholders.

Soft Skills

• Hit the ground running.
• Strong communication skills (written & verbal), interacting with Product, Design and Back-end Teams.
• Sense of urgency.
• Proactive to find solutions to new problems.
• High accountability.

Working Conditions

• Work in a standard office-based environment, with standard hours