Senior Big Data Engineer - Real-Time Data & Spark Expert

Aptonet; México; On-site; MXN 700,000 - 1,100,000; Full time

Job summary

Aptonet seeks a Big Data Engineer to design, build, and optimize high-velocity data pipelines. You will collaborate with data scientists and stakeholders to support data needs, and implement streaming solutions using Kafka, Spark Streaming, and Databricks.

You will deploy data platforms on Azure or AWS, manage serverless services, and work with Delta Lake, Unity Catalog, and NoSQL databases. The role emphasizes real-time processing and scalable architectures.

Qualifications

• Hands-on experience with high-velocity, high-volume stream processing: Kafka and Spark Streaming.
• Experience with real-time data processing using Spark Structured Streaming and Kafka.
• Troubleshooting and tuning Spark applications.
• Hands-on Python and/or Scala (PySpark/Scala-Spark).
• Experience with Databricks and building data pipelines.
• Deploy data platforms on Azure or AWS with serverless services like S3, Kinesis/MSK, Lambda, Glue.
• Experience with Delta Lake, Unity Catalog and Delta Live Tables.
• Knowledge of multiple data formats (JSON, XML, CSV) and NoSQL (HBASE, Cassandra).

Responsibilities

• Design and implement streaming data solutions using Kafka and Spark Streaming.
• Develop real-time data processing using Spark Structured Streaming and Kafka.
• Optimize Spark applications for peak performance.
• Work with Python and/or Scala for data processing (PySpark/Scala-Spark).
• Utilize Databricks for cloud-based big data solutions.
• Ingest data from JSON, XML, CSV formats and manage Delta Lake.
• Deploy data platforms on Azure or AWS and manage serverless components.
• Operate messaging platforms like Kafka, MSK, and IBM MQ Series.

Skills

Stream processing

Spark Streaming

PySpark

Scala-Spark

Databricks

Delta Lake

Unix/Linux scripting

Tools

Apache Kafka

Spark Structured Streaming

Azure

AWS

S3

Kinesis

MSK

Lambda

Glue

TIBCO EMS

IBM MQ Series

Delta Live Tables

Unity Catalog

Delta Lake UI

Kudu/Impala

Job description

Aptonet seeks a Big Data Engineer to design, build, and optimize high-velocity data pipelines. You will collaborate with data scientists and stakeholders to support data needs, and implement streaming solutions using Kafka, Spark Streaming, and Databricks.

You will deploy data platforms on Azure or AWS, manage serverless services, and work with Delta Lake, Unity Catalog, and NoSQL databases. The role emphasizes real-time processing and scalable architectures.