Big Data Specialist

Pyramid Consulting, Inc; Mexico; On-site; PHP 5,618,000 - 8,115,000; Full time

Job summary

Pyramid Consulting, Inc. is seeking an experienced Big Data Lead to design, develop, and support scalable data engineering solutions in a modern cloud-based environment.

You will work across data integration, ETL/ELT, and data pipelines to enable reliable analytics and decision making. The role emphasizes distributed data processing, SQL proficiency, and programming with Python/Java/Scala, plus hands-on experience with cloud platforms and data lake architectures.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field; equivalent practical experience may be considered.
• 5+ years of experience developing and supporting data integration, ETL/ELT, or data engineering solutions.
• Proven experience working with large data volumes in distributed or cloud-based environments; strong SQL skills.

Responsibilities

• Design, develop, and support scalable data integration, ETL/ELT, and data engineering solutions.
• Process and manage large data volumes across distributed and cloud-based environments.
• Develop efficient data pipelines using SQL and programming languages such as Python, Java, Scala, or Spark SQL.
• Build and manage data workflows using orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Step Functions.
• Design resilient and fault-tolerant pipelines with logging, monitoring, retry mechanisms, idempotency, and failure recovery.
• Work with relational and non-relational databases, data warehouses, and data lake architectures.
• Troubleshoot data pipeline and integration issues and implement effective solutions.
• Collaborate with engineering, application, and business teams to deliver reliable data solutions.
• Contribute to data architecture, modeling, governance, metadata management, and master data management initiatives.
• Support containerized deployments, CI/CD pipelines, and Infrastructure as Code practices.

Skills

Data integration

ETL/ELT

Distributed data processing

Cloud platforms

SQL

Programming (Python/Java/Scala)

Education

Bachelor's degree in Computer Science, Engineering, Information Systems, or related field

Job description

We are looking for an experienced Big Data Lead to design, develop, and support scalable data engineering solutions in a modern cloud-based environment. The ideal candidate will have strong experience with data integration, ETL/ELT, distributed data processing, cloud platforms, SQL, and programming, along with a solid understanding of data lakes, data warehouses, and resilient data pipelines.

Key Responsibilities

• Design, develop, and support scalable data integration, ETL/ELT, and data engineering solutions.
• Process and manage large volumes of data across distributed and cloud-based environments.
• Develop efficient data pipelines using SQL and programming languages such as Python, Java, Scala, or Spark SQL.
• Build and manage data workflows using orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Step Functions.
• Design resilient and fault-tolerant pipelines with logging, monitoring, retry mechanisms, idempotency, and failure recovery.
• Work with relational and non-relational databases, data warehouses, and data lake architectures.
• Troubleshoot data pipeline and integration issues and implement effective solutions.
• Collaborate with engineering, application, and business teams to deliver reliable data solutions.
• Contribute to data architecture, modeling, governance, metadata management, and master data management initiatives.
• Support containerized deployments, CI/CD pipelines, and Infrastructure as Code practices.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field; equivalent practical experience may be considered.
• 5+ years of experience developing and supporting data integration, ETL/ELT, or data engineering solutions.
• Proven experience working with large data volumes in distributed or cloud-based environments.
• Strong proficiency in SQL.
• Strong programming experience with at least one of the following:
• Python
• Java
• Scala
• Spark SQL
• Experience with workflow and orchestration technologies such as:
• AWS Step Functions
• Working knowledge of relational and non-relational databases.
• Experience with data warehouses and data lake architectures.
• Experience designing reliable production data pipelines with monitoring, retry logic, logging, and failure recovery.
• Strong analytical, troubleshooting, communication, and collaboration skills.

Preferred Qualifications

• Hands-on experience with AWS cloud data platforms.
• Experience with AWS Lambda and Apache Kafka.
• Knowledge of data modeling, metadata management, data governance, and Master Data Management (MDM).
• Experience with Docker/containerization and deployment technologies.
• Experience with CI/CD pipelines and Infrastructure as Code (IaC).
• Experience working in financial services, healthcare, regulated industries, or other highly controlled environments.