Senior Databricks Engineer: Flexible Data Pipelines

AgileEngine, LLC.; Pereira; Híbrido; COP 100.000.000 - 150.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flextime

Competitive USD-based compensation

Professional growth & mentorship

Project variety with Fortune 500 and领先

Descripción de la vacante

AgileEngine, LLC. is seeking a Senior Data Engineer to design, build, and operate batch and streaming pipelines on Databricks using PySpark and Delta Lake. The role focuses on migrating legacy workloads to a governed Lakehouse and enabling analytics and AI use cases.

The ideal candidate has strong SQL and Python skills, experience with Unity Catalog governance, and a track record of delivering scalable data solutions across cloud architectures.

Formación

• 3+ years of professional data engineering experience with Spark and cloud data architectures.
• Strong SQL and Python skills with data modeling across dimensional and Lakehouse patterns.
• Experience with Databricks, Spark (PySpark), and Delta Lake.
• Experience with streaming (Structured Streaming, Kafka or Event Hubs).
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience migrating legacy ETL workloads to Lakehouse and ensuring data parity.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience with dbt or equivalent transformation framework.
• Knowledge of secure coding practices and security best practices.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with minimal disruption.
• Use Claude or GitHub Copilot for code scaffolding, tests, docs, and prototypes.
• Write clean Python and SQL; ensure high quality via code reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance controls using Unity Catalog.
• Debug, troubleshoot, and resolve pipeline failures and production incidents.

Conocimientos

Apache Spark

Cloud data architectures

SQL

Python

Structured Streaming

Databricks

Delta Lake

Data modeling

Unity Catalog

Data governance

dbt

Herramientas

Databricks

Airflow

Azure Data Factory

Descripción del empleo

AgileEngine, LLC. is seeking a Senior Data Engineer to design, build, and operate batch and streaming pipelines on Databricks using PySpark and Delta Lake. The role focuses on migrating legacy workloads to a governed Lakehouse and enabling analytics and AI use cases.

The ideal candidate has strong SQL and Python skills, experience with Unity Catalog governance, and a track record of delivering scalable data solutions across cloud architectures.