Senior Databricks Data Engineer — Lakehouse Pipelines (Remote)

AgileEngine; Centrosur; Presencial; COP 282.672.000 - 408.304.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth without limits

Competitive compensation

Remote work

Meaningful projects

Collaborative culture

Well-being programs

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a legacy data warehouse into a governed Databricks Lakehouse. You will design and run batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting, and ML.

Join a team leveraging AI tools like Claude and GitHub Copilot, focusing on data quality, governance, and scalable production platforms. This role supports remote work with a strong collaboration culture.

Formación

• 4+ years of professional experience in data engineering with Spark expertise.
• Strong hands-on experience building data pipelines with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with strong data modeling skills.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or data hygiene projects.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics, reporting, and ML.
• Migrate legacy ETL workloads onto the Lakehouse with minimal business disruption.
• Leverage Claude or GitHub Copilot to accelerate development, tests, docs, and prototyping.
• Write clean Python and SQL; enforce high quality via reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost (partitioning, clustering, caching).
• Implement data quality, lineage, and governance with Unity Catalog and tests.
• Debug and resolve pipeline failures and production incidents.
• Participate in Agile or product-centric delivery with sprint planning and retrospectives.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance.

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a legacy data warehouse into a governed Databricks Lakehouse. You will design and run batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting, and ML.

Join a team leveraging AI tools like Claude and GitHub Copilot, focusing on data quality, governance, and scalable production platforms. This role supports remote work with a strong collaboration culture.