Remote Senior Databricks Data Engineer — Lakehouse & AI

AgileEngine, LLC.; Centrosur; Presencial; COP 385.493.000 - 578.239.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Competitive compensation

Remote-friendly

Challenging projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine, LLC. is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake while applying a medallion architecture for analytics and ML workloads.

You will collaborate with cross-functional teams, optimize Spark jobs for performance and cost, and drive governance and data quality across the platform using Unity Catalog and automated checks.

Formación

• 4+ years of professional experience in data engineering.
• Hands-on building data pipelines with Databricks, Spark (PySpark) and Delta Lake.
• Advanced SQL and Python with strong data modeling.
• Experience with streaming ingestion (Structured Streaming, Auto Loader, Kafka, Event Hubs).
• Workflow orchestration with Databricks Workflows, Airflow or Azure Data Factory.
• Experience migrating legacy ETL workloads and data warehouses.
• Strong problem solving, collaboration and communication; mentoring preferred.
• Familiarity with Unity Catalog, data governance and PII handling.
• Experience with dbt or equivalent transformation framework.
• Familiarity with secure coding standards and security best practices.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks.
• Model and maintain a medallion architecture for analytics and ML consumers.
• Migrate legacy workloads to the Lakehouse with minimal disruption.
• Use Claude or Copilot to accelerate development, tests, and docs.
• Write Python and SQL, and conduct code reviews and documentation.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog.
• Debug and resolve pipeline failures and production incidents.
• Collaborate in Agile rituals and with DevOps, platform, and analytics teams.

Conocimientos

Apache Spark (PySpark)

Delta Lake

SQL

Python

Data modeling

Herramientas

Databricks

Airflow

Azure Data Factory

dbt

Terraform

CI/CD (Azure DevOps)

Descripción del empleo

AgileEngine, LLC. is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake while applying a medallion architecture for analytics and ML workloads.

You will collaborate with cross-functional teams, optimize Spark jobs for performance and cost, and drive governance and data quality across the platform using Unity Catalog and automated checks.