Senior Databricks Data Engineer — Remote, Flexible Hours

AgileEngine, LLC.; Monterrey; Presencial; MXN 700.000 - 1.100.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Growth

Compensation

Remote work

Modern projects

Collaborative culture

Well-being

Descripción de la vacante

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

Use AI tools like Claude and Copilot to speed up development. Responsibilities include migrating legacy workloads, writing clean Python/SQL, and ensuring data quality and governance with Unity Catalog.

Formación

• 4+ years of professional experience in data engineering with Spark and cloud data architectures.
• Hands-on experience building data pipelines with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with strong data modeling in Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration Databricks Workflows, Airflow, or Azure Data Factory.
• Experience with legacy platform migrations and ETL modernization.
• Strong problem-solving, collaboration, and communication, including mentoring.
• Familiarity with Unity Catalog, data governance, and PII handling.
• Experience with dbt or similar transformation frameworks.
• Knowledge of secure coding standards and security best practices.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks with PySpark, Delta Lake, and Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics, reporting, and ML.
• Migrate legacy ETL workloads to the Lakehouse with minimal disruption and ensured data parity.
• Leverage Claude or Github Copilot to accelerate development, tests, docs, and prototyping.
• Write clean Python and SQL, with strong coding discipline and review processes.
• Optimize Spark jobs and Delta tables for performance and cost through partitioning and clustering.
• Implement data quality, lineage, and governance using Unity Catalog and validations.
• Debug and resolve pipeline failures and production incidents promptly.
• Collaborate in Agile delivery, including sprint planning and retrospectives.
• Work with DevOps, platform, and analytics engineers on observability, security, and compliance.

Conocimientos

Mentoring junior engineers

Strong communication

Problem solving

Team collaboration

Herramientas

Databricks

Apache Spark (PySpark)

Delta Lake

SQL

Python

Structured Streaming

Auto Loader

Kafka

Event Hubs

Databricks Workflows

Airflow

Azure Data Factory

Unity Catalog

dbt

Descripción del empleo

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

Use AI tools like Claude and Copilot to speed up development. Responsibilities include migrating legacy workloads, writing clean Python/SQL, and ensuring data quality and governance with Unity Catalog.