Senior Databricks Lakehouse Engineer | Remote & Flexible

AgileEngine; Pereira; Presencial; COP 120.000.000 - 180.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility: 100% remote

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, and leverage AI tools to accelerate development.

You will build a medallion architecture, migrate legacy workloads, and collaborate with DevOps and analytics teams to ensure governance and data quality across projects. This is a fully remote role with global collaboration.

Formación

• 4+ years of professional data engineering experience with Spark and cloud data architectures.
• Strong Python and SQL skills with data modeling in Lakehouse contexts.
• Experience building batch/streaming pipelines and data quality controls.

Responsabilidades

• Design and operate batch and streaming data pipelines on Databricks with PySpark and Delta Lake.
• Maintain medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to lakehouse with data parity and minimal disruption.
• Utilize Claude or Copilot to accelerate development and testing.
• Write clean Python/SQL and enforce code reviews and documentation.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data governance using Unity Catalog and automated checks.
• Debug pipelines and resolve production incidents.
• Participate in Agile rituals and collaborate with cross-functional teams.
• Work with DevOps and analytics engineers on observability and security.

Conocimientos

Spark/PySpark

Data modeling

Python

SQL

Coding best practices

Team collaboration

Herramientas

Databricks

Delta Lake

Airflow

Azure Data Factory

Cloud platforms

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, and leverage AI tools to accelerate development.

You will build a medallion architecture, migrate legacy workloads, and collaborate with DevOps and analytics teams to ensure governance and data quality across projects. This is a fully remote role with global collaboration.