Senior Databricks Data Engineer - Remote, Growth & Impact

AgileEngine, LLC.; Metropolitana; Presencial; COP 120.000.000 - 180.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth opportunities

Remote work

Learning budget

Competitive pay

Work with global teams

Well-being programs

Descripción de la vacante

AgileEngine, LLC is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will design and operate batch and streaming pipelines using PySpark, Delta Lake, and Databricks Workflows.

You’ll collaborate with cross-functional teams, mentor junior engineers, and apply data governance, quality, and security best practices in a fast-growing, remote-friendly environment.

Formación

• 4+ years of professional data engineering experience with Spark and cloud data architectures.
• Hands-on experience with Databricks, PySpark, and Delta Lake.
• Advanced SQL and Python with strong data modeling skills.
• Experience enabling streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflows: Databricks Workflows, Airflow, or Azure Data Factory.
• Experience with legacy platform migrations and ETL modernization.
• Familiar with Unity Catalog, data governance, and PII handling.
• Experience with dbt or an equivalent transformation framework.
• Secure coding practices and collaboration across teams.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics and ML consumers.
• Migrate legacy ETL workloads to a Lakehouse with data parity and minimal disruption.
• Use Claude or Github Copilot to accelerate development, tests, and docs.
• Write clean Python and SQL; ensure high-quality code via reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance checks with Unity Catalog.
• Debug and resolve pipeline failures and production incidents.
• Participate in Agile delivery practices and collaborate across teams.
• Collaborate with DevOps and analytics engineers on observability, security, and compliance.

Conocimientos

Databricks

Apache Spark

PySpark

SQL

Python

Data modeling

Streaming pipelines

Mentoring

Communication

Unity Catalog

Herramientas

Databricks Workflows

Airflow

Azure Data Factory

dbt

Terraform

PostgreSQL

CloudWatch

Dynatrace

Descripción del empleo

AgileEngine, LLC is seeking a Senior Data Engineer to modernize a 15-year data warehouse into a governed Databricks Lakehouse. You will design and operate batch and streaming pipelines using PySpark, Delta Lake, and Databricks Workflows.

You’ll collaborate with cross-functional teams, mentor junior engineers, and apply data governance, quality, and security best practices in a fast-growing, remote-friendly environment.