Senior Data Engineer - Remote, Databricks Lakehouse Architect

INGEPSY; Metropolitana; Presencial; COP 372.821.000 - 559.232.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility

Meaningful, modern projects

Collaborative culture

Well-being & support

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

You will also leverage AI tools like Claude and GitHub Copilot to speed up development, tests, and documentation. Strong Python and SQL skills are required.

Formación

• 4+ years of data engineering experience with Spark
• Strong SQL and Python skills
• Experience with Databricks and Delta Lake
• Knowledge of data lakehouse concepts

Responsabilidades

• Design, build, and operate batch and streaming pipelines on Databricks with PySpark and Delta Lake.
• Maintain a medallion architecture (bronze/silver/gold) for analytics and ML consumers.
• Migrate legacy ETL and data warehouse workloads to the Lakehouse with minimal disruption.
• Use Claude or GitHub Copilot to accelerate coding, tests, and documentation.
• Write clean, well-tested Python and SQL; enforce coding standards.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, governance with Unity Catalog and automated checks.
• Debug production pipelines and resolve data defects.
• Collaborate in Agile delivery with DevOps, platform, and analytics engineers.

Conocimientos

Data engineering

Apache Spark

PySpark

SQL

Python

Cloud data architectures

Herramientas

Databricks

Delta Lake

Databricks Workflows

Airflow

GitHub Copilot

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

You will also leverage AI tools like Claude and GitHub Copilot to speed up development, tests, and documentation. Strong Python and SQL skills are required.