Mid Data Engineer - Databricks Lakehouse (Remote)

INGEPSY; Metropolitana; Presencial; COP 279.616.000 - 466.027.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Remote work 100%

Learning budget

Competitive compensation

Flexible hours

Descripción de la vacante

AgileEngine is seeking a Middle Data Engineer to modernize a data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

You will use AI tools like Claude and GitHub Copilot to speed up development, and collaborate with cross-functional teams to ensure data quality and governance. This is a remote, globally distributed role.

Formación

• 3+ years of professional data engineering experience.
• Hands-on with Databricks, Apache Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with data modeling skills.
• Experience with streaming ingestion and orchestration tools.
• Familiarity with Unity Catalog and data governance.
• dbt or equivalent transformation framework experience.
• Knowledge of secure coding standards and production data platforms.

Responsabilidades

• Design and operate batch and streaming pipelines on Databricks.
• Maintain medallion architecture (bronze/silver/gold) for analytics.
• Migrate legacy ETL workloads to a Lakehouse with minimal disruption.
• Leverage AI tools to accelerate development and testing.
• Write clean Python and SQL and enforce code quality.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance controls.
• Troubleshoot pipelines and production incidents.
• Collaborate with DevOps and analytics engineers on observability and security.

Conocimientos

Apache Spark (PySpark)

Databricks

Delta Lake

SQL

Python

Structured Streaming

Kafka

Airflow

Databricks Workflows

Azure Data Factory

Unity Catalog

dbt

Security best practices

Cloud data architectures

Herramientas

Terraform

PostgreSQL

Dynatrace / CloudWatch

Descripción del empleo

AgileEngine is seeking a Middle Data Engineer to modernize a data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

You will use AI tools like Claude and GitHub Copilot to speed up development, and collaborate with cross-functional teams to ensure data quality and governance. This is a remote, globally distributed role.