Senior Databricks Data Engineer - Remote Lakehouse Pipelines

AgileEngine, LLC.; Perímetro Urbano Barranquilla; Presencial; COP 353.368.000 - 481.866.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

100% remote with flexible hours

Annual learning budget

Mentorship and TechTalks

Global collaboration

Descripción de la vacante

AgileEngine, LLC. is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, using a medallion architecture across bronze, silver, and gold layers. AI tools like Claude and Copilot speed up development.

Design, implement, and optimize data pipelines while ensuring governance, data quality, and security. Join a people-first culture with remote-friendly arrangements.

Formación

• 4+ years of professional data engineering experience with Spark and cloud data architectures.
• Hands-on with Databricks, PySpark, and Delta Lake for batch/streaming pipelines.
• Advanced SQL and Python with data modeling across Lakehouse patterns.
• Experience with Structured Streaming, Auto Loader, Kafka or Event Hubs.
• Orchestrating data workflows with Databricks Workflows, Airflow or Azure Data Factory.
• Knowledge of data hygiene, migration, governance, and Unity Catalog.
• Familiarity with dbt or equivalent transformation framework.
• Secure coding practices and strong collaboration with teams.

Responsabilidades

• Design, build, and run batch and streaming pipelines on Databricks with PySpark and Delta Lake.
• Model and maintain medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to Lakehouse with minimal business disruption.
• Leverage Claude or Copilot to accelerate development, tests, and docs.
• Write clean Python and SQL; enforce quality through reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance controls via Unity Catalog.
• Troubleshoot and resolve pipeline failures and data defects in production.
• Collaborate in Agile delivery, sprint planning, and cross-functional teams.
• Coordinate with DevOps, platform, and analytics for observability and security.

Conocimientos

Apache Spark

Databricks

Python

SQL

Data modeling

Delta Lake

ETL

Streaming pipelines

Airflow

GitHub Copilot

Herramientas

Databricks Workflows

Azure Data Factory

Airflow

Unity Catalog

Descripción del empleo

AgileEngine, LLC. is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, using a medallion architecture across bronze, silver, and gold layers. AI tools like Claude and Copilot speed up development.

Design, implement, and optimize data pipelines while ensuring governance, data quality, and security. Join a people-first culture with remote-friendly arrangements.