Data Engineer - Remote, Databricks Lakehouse & Pipelines

AgileEngine, LLC.; Metropolitana; Presencial; COP 90.000.000 - 140.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Flexibility: 100% remote

Descripción de la vacante

AgileEngine, LLC. is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, following a bronze/silver/gold medallion architecture for analytics and ML.

You will migrate legacy ETL workloads, ensure data parity, and leverage AI tooling (Claude, Copilot) to accelerate development. Strong SQL, Python, and collaboration across teams are essential.

Formación

• 3+ years of professional data engineering experience.
• Proficient with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python skills.
• Experience with streaming ingestion (Structured Streaming, Auto Loader, Kafka, Event Hubs).
• Experience with data workflows (Databricks Workflows, Airflow, Azure Data Factory).
• Migrations of legacy ETL/data warehouse workloads.
• Familiarity with Unity Catalog and data governance.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks with PySpark and Delta Lake.
• Model and maintain medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to Lakehouse with validated data parity.
• Use Claude or Copilot to accelerate development, write tests and docs.
• Write clean Python and SQL; conduct code reviews and documentation.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, governance, and validation checks.
• Troubleshoot pipelines, data defects, and production incidents.
• Collaborate with DevOps, platform, and analytics teams on observability and security.

Conocimientos

Apache Spark (PySpark)

Delta Lake

SQL

Python

Structured Streaming

Databricks Workflows

Airflow

Azure Data Factory

dbt

Unity Catalog

Herramientas

Databricks

Apache Spark (PySpark)

Delta Lake

Kafka

Airflow

Azure Data Factory

dbt

Descripción del empleo

AgileEngine, LLC. is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming pipelines with PySpark and Delta Lake, following a bronze/silver/gold medallion architecture for analytics and ML.

You will migrate legacy ETL workloads, ensure data parity, and leverage AI tooling (Claude, Copilot) to accelerate development. Strong SQL, Python, and collaboration across teams are essential.