Databricks Engineer

Apex Systems; Mexico; On-site; MXN 900,000 - 1,200,000; Full time

Job summary

Apex Systems is seeking a Senior Databricks Engineer to lead the design and implementation of a modern Databricks Lakehouse architecture. You will migrate legacy data solutions into scalable, governed pipelines using Databricks, PySpark, and Delta Lake.

You will define and build Bronze, Silver, and Gold data layers to enable both batch and near real-time processing, while guiding and mentoring junior engineers and collaborating with data teams and stakeholders.

Qualifications

• 6+ years of experience in data engineering, with 3+ years of hands-on Databricks experience.
• Strong expertise in PySpark, Spark SQL, and Delta Lake.
• Proven experience migrating legacy data platforms such as SQL Server, SSIS, and ADF.

Responsibilities

• Architect and implement end-to-end data solutions within the Databricks Lakehouse environment.
• Design and build ingestion, transformation, and serving pipelines using PySpark, Spark SQL, and Delta Lake.
• Establish medallion architecture practices across Bronze, Silver, and Gold layers.

Skills

Databricks experience

Data engineering

Leadership

Mentoring

Education

Bachelor’s degree in Computer Science, Engineering, Information Systems

Tools

PySpark

Spark SQL

Delta Lake

Azure SQL

SSIS

ADF

Unity Catalog

Job description

We are seeking a Senior Databricks Engineer to lead the design and implementation of a modern Databricks Lakehouse architecture. This role is focused on migrating legacy data solutions into scalable, governed, and performance-optimized pipelines using Databricks, PySpark, and Delta Lake.

The engineer will play a key role in defining and building Bronze, Silver, and Gold data layers, enabling both batch and near real-time processing, and improving the reliability and usability of enterprise data platforms. This position will work closely with data engineers, architects, and stakeholders while providing technical leadership and guidance to more junior team members.

Key Responsibilities

Architecture & Engineering

• Architect and implement end-to-end data solutions within the Databricks Lakehouse environment
• Design and build ingestion, transformation, and serving pipelines using PySpark, Spark SQL, and Delta Lake
• Establish and enforce medallion architecture practices across Bronze, Silver, and Gold layer

Migration & Modernizatio

• nLead the migration of legacy Azure SQL, SSIS, and ADF pipelines into Databricks-native workflows
• Convert existing T-SQL and ETL logic into scalable Spark-based processing
• Support modernisation of data platforms with a focus on governance, scalability, and maintainability

Streaming, Governance & Optimization

• Implement Auto Loader, Delta Lake Change Data Feed (CDF), and streaming/batch ingestion patterns
• Define and enforce governance using Unity Catalog, including RBAC and data lineage
• Optimise performance through Photon, partitioning, Z-Order indexing, and cluster tuning

Validation & Technical Leadership

• Guide validation and reconciliation strategies between legacy SQL outputs and Delta-based pipelines
• Perform code reviews and promote engineering best practices
• Mentor mid-level and junior engineers across design, development, and troubleshooting activities

Required Experience & Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or equivalent experience
• 6+ years of experience in data engineering, with 3+ years of hands-on Databricks experience
• Strong expertise in PySpark, Spark SQL, and Delta Lake
• Proven experience migrating legacy data platforms such as SQL Server, SSIS, and ADF
• Strong understanding of ETL/ELT design, data modeling, and pipeline architecture
• Hands-on experience with the Azure ecosystem, including ADLS, ADF, and related services
• Strong knowledge of performance tuning techniques for distributed data processing
• Experience leading technical implementation efforts and supporting junior engineers

Preferred Qualifications

• Experience with Unity Catalog, Lakehouse Federation, or CDC frameworks
• Databricks certifications (Associate or Professional level)
• Exposure to large-scale regulated datasets such as healthcare, claims, or financial data
• Experience with both batch and streaming data pipelines
• Familiarity with enterprise-grade data governance and lineage practices