Sr Data Engineer/ 100% Remote in Mexico

Pyramid Consulting, Inc; Mexico; On-site; PHP 3,317,000 - 4,792,000; Full time

Job summary

Pyramid Consulting, Inc. is seeking a Data Engineer to design, build, and maintain scalable cloud-native data pipelines and infrastructure.

You will work with analytics, BI, and data science teams to deliver reliable data solutions using modern cloud, big data, and API-driven technologies. The role requires strong SQL, Python and Spark skills, and hands-on experience with Azure Databricks, Delta Lake, and API development.

Qualifications

• Strong SQL proficiency with relational databases.
• Proficient in Python, PySpark, Java or Scala.
• Experience with Azure cloud services and Databricks Lakehouse.
• Knowledge of real-time data ingestion and streaming tech such as Kafka or Event Hubs.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines for batch and real-time processing.
• Develop and optimize data models, Delta Tables, and Lakehouse architectures.
• Create RESTful APIs and data services for seamless data exchange.
• Implement real-time ingestion frameworks and streaming architectures.
• Ensure data quality, governance, security, and compliance across platforms.

Skills

Analytical thinking

Python programming

APIs & microservices

Education

Bachelor's degree in Computer Science or similar

Tools

Azure Cloud

Databricks

Delta Lake

Delta Tables

Spark

OpenShift

Kubernetes

Docker

Airflow

Azure Data Factory

Event Hubs

Synapse Analytics

Job description

Job Type: Full-time based Job Opportunity

Location: 100% Remote in Mexico

Job Description:

The Data Engineer is responsible for designing, building, and maintaining scalable, cloud-native data pipelines and data infrastructure that support analytics, reporting, business intelligence, and real-time data processing. This role ensures that data is accessible, reliable, secure, and optimized for performance across enterprise platforms. The Data Engineer collaborates with business stakeholders, analysts, data scientists, and application teams to deliver high-quality data solutions using modern cloud, big data, and API-driven technologies.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for batch and real-time data processing.
• Build and optimize data models, Delta Tables, and Lakehouse architectures to support analytics and reporting.
• Develop and integrate RESTful APIs and data services to facilitate seamless data exchange across enterprise systems.
• Implement real-time and high-frequency data ingestion frameworks using streaming technologies and event-driven architectures.
• Design and manage cloud-native data solutions leveraging Azure services including Azure Data Factory, Azure Databricks, ADLS, Event Hubs, and Synapse Analytics.
• Develop and optimize Databricks Spark applications for large-scale data transformation and processing.
• Ensure data quality, governance, security, and compliance across data platforms.
• Collaborate with data scientists, analysts, application teams, and business stakeholders to deliver scalable data solutions.
• Troubleshoot, monitor, and optimize pipeline performance and data platform reliability.
• Support DataOps and CI/CD practices for data pipeline deployment and automation.

Required Skills & Qualifications

• Strong proficiency in SQL and relational databases such as Oracle, SQL Server, and MySQL.
• Strong programming skills in Python, PySpark, PL/SQL, Java, or Scala.
• Hands-on experience with Azure Cloud technologies:
• Experience with Databricks Lakehouse architecture, Delta Lake, and Delta Tables.
• Expertise in API development, API integration, RESTful services, and microservices architecture.
• Experience processing high-volume and high-frequency data with low-latency requirements.
• Strong knowledge of real-time data ingestion and streaming technologies such as Kafka, Azure Event Hubs, or Kinesis.
• Experience with Spark, Hadoop, and distributed data processing frameworks.
• Hands-on experience with OpenShift, Kubernetes, Docker, and containerized deployments.
• Experience with workflow orchestration tools such as Apache Airflow and Azure Data Factory.
• Understanding of data governance, data security, and compliance best practices.