Data Engineer

Pyramid Consulting, Inc; Región Centro; Presencial; MXN 900.000 - 1.500.000; Jornada completa

Descripción de la vacante

Pyramid Consulting, Inc is seeking a Senior Data Engineer to help build and scale data analytics products on on-premise and cloud platforms such as Databricks and Hadoop. You will design robust pipelines, optimize performance, and partner with Product Managers to translate use cases into data modules.

The role emphasizes Python, Spark, SQL, and tools like Airflow and NiFi, with an eye toward CI/CD, testing, and maintainable code.

Formación

• Strong hands-on experience in data engineering building production-grade pipelines on Hadoop and cloud platforms (Hadoop ecosystem and cloud data platforms - Databricks).
• Proficiency with Python, Spark, Hadoop tools (Hive, Impala, Airflow, NiFi) and SQL for Big Data products.
• Experience with cloud data platforms such as Databricks.
• Familiarity with orchestration tools like Airflow, NiFi, or Talend.
• Knowledge of DevOps/CI-CD practices: Git, automated testing, release pipelines, and observability.
• Strong problem-solving and communication skills for technical and non-technical stakeholders.
• Experience developing Java-based applications is an added advantage.

Responsabilidades

• Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale.
• Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
• Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as Cloudera and Databricks.
• Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns.
• Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.
• Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features.

Conocimientos

Big data pipelines

Python

Spark

SQL

Problem-solving

Git

CI/CD

Java

Herramientas

Databricks

Hive

Impala

Airflow

NiFi

Talend

Descripción del empleo

Overview We are looking for a Senior Data Engineer who will help in implementing data analytics products using on-prem and cloud data platforms.

Position Responsibilities:

• Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.
• Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
• Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.
• Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns
• Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.
• Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features

Ideal Candidate Qualifications:

• Strong hands-on experience in data engineering building production-grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).
• High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.
• Hands-on experience with cloud data platforms such as databricks, Client (databricks preferred)
• Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.
• Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.
• Strong problem-solving skills with the ability to debug complex data issues and communicate clearly with technical and non-technical stakeholders.
• Experience developing Java based applications is an added advantage.