Databricks Data Engineer — Flexible, High-Impact Analytics

AgileEngine, LLC.; Pereira; Híbrido; COP 280.260.000 - 404.820.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive USD-based compensation

Exciting projects with top-tier brands

Flextime

Descripción de la vacante

AgileEngine, LLC. is seeking a Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse, modeling a medallion architecture for analytics and AI use cases.

The role emphasizes strong SQL, Python, and Unity Catalog governance, with collaboration across DevOps, platform, and analytics teams in an Agile environment. Flexible work options are available.

Formación

• 4+ years of professional data engineering experience.
• Expertise with Apache Spark, PySpark, Delta Lake, and Databricks.
• Strong SQL and Python with data modeling across Lakehouse patterns.
• Experience migrating ETL workloads to Lakehouse.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and machine learning consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
• Use Claude or GitHub Copilot as a development accelerator, generating code scaffolding, writing and reviewing tests, creating documentation, and prototyping solutions.
• Write clean, well-tested Python and SQL; maintain high standards through code review and documentation.
• Optimize Spark jobs and Delta tables for performance and cost, including partitioning, clustering, caching, and cluster sizing.
• Implement data quality, lineage, and governance controls using Unity Catalog and automated validation checks.
• Debug, troubleshoot, and resolve pipeline failures, data defects, and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices.
• Participate in Agile or product-centric delivery practices, including sprint planning and retrospectives.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices.

Conocimientos

Data engineering

Apache Spark

SQL

Python

Streaming ingestion

Herramientas

Databricks

Apache Spark (PySpark)

Delta Lake

Unity Catalog

Airflow

Azure Data Factory

dbt

Descripción del empleo

AgileEngine, LLC. is seeking a Data Engineer to design and operate batch and streaming data pipelines on Databricks using PySpark and Delta Lake. You will migrate legacy ETL workloads to a governed Lakehouse, modeling a medallion architecture for analytics and AI use cases.

The role emphasizes strong SQL, Python, and Unity Catalog governance, with collaboration across DevOps, platform, and analytics teams in an Agile environment. Flexible work options are available.