Senior Databricks Data Engineer | Remote & Growth-Focused

AgileEngine, LLC.; Medellín; Presencial; COP 130.000.000 - 170.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Remote work with flexible hours

Modern projects with global teams

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine, LLC. is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. Responsibilities include migrating legacy ETL workloads, optimizing Spark jobs, and implementing data governance with Unity Catalog.

Formación

• 4+ years of professional experience in data engineering.
• Hands-on experience with Databricks, Apache Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with strong data modeling skills.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or data hygiene when porting old systems.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience with dbt or an equivalent transformation framework.
• Familiarity with secure coding standards and industry security best practices.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) serving analytics and ML consumers.
• Migrate legacy ETL workloads to the Lakehouse with minimal business disruption.
• Leverage Claude or Copilot to accelerate development, tests, and documentation.
• Write and review clean Python and SQL with strong testing standards.
• Tune Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog.
• Debug and resolve production data issues and pipeline failures.
• Collaborate in Agile or product-centric delivery plans with cross-functional teams.
• Work with DevOps and analytics engineers on observability, security, and compliance.

Conocimientos

Databricks

Apache Spark (PySpark)

SQL

Python

Structured Streaming

Data modeling

Azure Data Factory

Airflow

dbt

Unity Catalog

Security best practices

Mentoring

Educación

Herramientas

Databricks Workflows

Airflow

Azure Data Factory

dbt

Unity Catalog

Descripción del empleo

AgileEngine, LLC. is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. Responsibilities include migrating legacy ETL workloads, optimizing Spark jobs, and implementing data governance with Unity Catalog.