Data Engineer

Pyramid Consulting, Inc; México; Presencial; MXN 350.000 - 600.000; Jornada completa

Descripción de la vacante

Pyramid Consulting, Inc. is seeking a Databricks Engineer to design, build, and optimize scalable data processing pipelines on the Azure Databricks platform.

The role collaborates with data engineers, analysts, and stakeholders to deliver high-performance data solutions across Azure cloud services. The ideal candidate has 3+ years in data engineering, strong PySpark and Spark skills, and a solid grasp of data governance and CI/CD for data workflows.

Formación

• Bachelor's degree in Computer Science, Data Engineering, or related field.
• 3+ years of experience in data engineering or big data technologies.
• Hands-on experience with Databricks, Apache Spark, and PySpark.
• Strong knowledge of SQL, Python, and data modeling principles.
• Experience with cloud platforms (AWS, Azure, or GCP) and their data services.
• Familiarity with Delta Lake, Lakehouse architecture, and data governance.
• Understanding of CI/CD pipelines and DevOps practices for data workflows.
• Excellent problem-solving and communication skills.

Responsabilidades

• Design, build, and maintain data pipelines and ETL processes using Databricks and Apache Spark.
• Optimize data workflows for performance, scalability, and cost efficiency.
• Implement data Lakehouse architecture and manage data ingestion from multiple sources.
• Collaborate with data scientists and analysts to enable advanced analytics and machine learning workloads.
• Ensure data quality, governance, and security across all data assets.
• Monitor and troubleshoot Databricks clusters, jobs, and workflows.
• Integrate Databricks with cloud services (AWS, Azure, or GCP) and other enterprise systems.
• Document processes, standards, and best practices for data engineering.

Conocimientos

Databricks

Apache Spark

PySpark

SQL

Python

Data Modeling

Cloud platforms

CI/CD for data pipelines

DevOps practices

Data governance

Communication

Educación

Bachelor's degree in Computer Science or related field

Herramientas

Databricks

Apache Spark

PySpark

Delta Lake

Azure Data Factory

Lakehouse architecture

Descripción del empleo

• We are seeking a skilled Databricks Engineer with strong experience in Azure Databricks and Python (PySpark) to design, build, and optimize scalable data processing pipelines.
• The ideal candidate will work closely with data engineers, analysts, and stakeholders to deliver high-performance data solutions on the Azure cloud platform.

Key Responsibilities

• Design, build, and maintain data pipelines and ETL processes using Databricks and Apache Spark.
• Optimize data workflows for performance, scalability, and cost efficiency.
• Implement data Lakehouse architecture and manage data ingestion from multiple sources.
• Collaborate with data scientists and analysts to enable advanced analytics and machine learning workloads.
• Ensure data quality, governance, and security across all data assets.
• Monitor and troubleshoot Databricks clusters, jobs, and workflows.
• Integrate Databricks with cloud services (AWS, Azure, or GCP) and other enterprise systems.
• Document processes, standards, and best practices for data engineering.

Required Skills & Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, or related field.
• 3+ years of experience in data engineering or big data technologies.
• Hands-on experience with Databricks, Apache Spark, and PySpark.
• Strong knowledge of SQL, Python, and data modeling principles.
• Experience with cloud platforms (AWS, Azure, or GCP) and their data services.
• Familiarity with Delta Lake, Lakehouse architecture, and data governance.
• Understanding of CI/CD pipelines and DevOps practices for data workflows.
• Excellent problem-solving and communication skills.

Nice to Have

• Experience with Delta Lake and Lakehouse architecture.
• Knowledge of Azure Data Factory (ADF) for orchestration.
• Experience with SQL and Spark SQL.
• Azure certifications related to data engineering.