Databricks Data Engineer - Remote-Friendly Lakehouse & Pipelines

AgileEngine, LLC.; Perímetro Urbano Barranquilla; Híbrido; COP 155.700.000 - 342.540.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Professional growth

Competitive compensation (USD-based)

A selection of exciting projects

Flextime

Descripción de la vacante

AgileEngine, LLC. is seeking a Data Engineer to design and operate batch and streaming pipelines on Databricks, utilizing PySpark and Delta Lake to migrate legacy workloads into a governed Lakehouse. You will model medallion architecture and enable analytics, reporting, and ML use cases.

The role requires strong SQL, Python, and experience with orchestration tools, data governance, and secure coding practices. Collaboration with cross-functional teams is essential for success.

Formación

• 4+ years of professional experience in data engineering with Spark and cloud architectures.
• Hands-on with Databricks, Spark PySpark, and Delta Lake.
• Advanced SQL and Python with strong data modeling skills.
• Experience with Structured Streaming, Auto Loader, Kafka or Event Hubs.
• Experience with orchestration (Databricks Workflows, Airflow or Azure Data Factory).
• Experience migrating legacy ETL or data warehousing, and data hygiene.
• Strong problem-solving, collaboration, and mentoring.
• Familiarity with Unity Catalog, data governance and PII handling.
• Experience with dbt or similar transformation framework.
• Secure coding practices.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal disruption.
• Write clean, well-tested Python and SQL; maintain code quality through reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog.
• Collaborate with DevOps and analytics engineers on observability and security.

Conocimientos

Apache Spark

Python

SQL

Data modeling

Mentoring juniors

Communication

Herramientas

Databricks

Delta Lake

Azure Data Factory

Airflow

Unity Catalog

dbt

Kafka

Descripción del empleo

AgileEngine, LLC. is seeking a Data Engineer to design and operate batch and streaming pipelines on Databricks, utilizing PySpark and Delta Lake to migrate legacy workloads into a governed Lakehouse. You will model medallion architecture and enable analytics, reporting, and ML use cases.

The role requires strong SQL, Python, and experience with orchestration tools, data governance, and secure coding practices. Collaboration with cross-functional teams is essential for success.