Real-Time Data Engineer: Kafka, Spark & Cloud Platforms

Aptonet; México; On-site; MXN 700,000 - 1,000,000; Full time

Job summary

Aptonet is seeking an experienced data engineering professional to design and implement high-velocity data streaming solutions using Apache Kafka and Spark Streaming. You will develop real-time processing with Spark Structured Streaming, troubleshoot Spark workloads, and work with Python/Scala on Databricks in a cloud environment.

Responsibilities include building ingestion pipelines, deploying data platforms on Azure or AWS, managing messaging systems like Kafka/MSK, and handling data formats

Qualifications

• Hands-on experience with high-velocity, high-volume stream processing: Kafka and Spark Streaming.
• Real-time data processing using Spark Structured Streaming and Kafka.
• Troubleshooting and tuning Spark applications for peak performance.
• Proficiency in Python and/or Scala (PySpark/Scala-Spark).
• Experience with Databricks.
• Experience building and deploying data ingestion pipelines and datasets.
• Experience deploying a data platform on Azure/AWS with serverless tech like S3, Kinesis/MSK, Lambda, Glue.
• Knowledge of messaging platforms like Kafka, MSK, TIBCO EMS, or IBM MQ Series.
• Experience with Databricks Notebooks, Delta Lake, and Unity Catalog.
• Experience with JSON/XML/CSV data ingestion and NoSQL databases like HBASE/Cassandra.

Responsibilities

• Design and implement data streaming solutions using Kafka and Spark Streaming.
• Develop real-time processing with Spark Structured Streaming and Kafka.
• Troubleshoot and optimize Spark applications for performance.
• Collaborate with Python/Scala data processing tasks (PySpark/Scala-Spark).
• Utilize Databricks for cloud-based big data solutions.
• Build and optimize big data ingestion pipelines and architectures.
• Deploy data platforms on Azure or AWS with serverless services (S3, Kinesis, Lambda, Glue).
• Manage messaging platforms like Kafka, MSK, TIBCO EMS, IBM MQ Series.
• Work with Databricks Notebooks, Delta Lake, and Unity Catalog.
• Ingest data from JSON, XML, and CSV formats.
• Work with NoSQL databases like HBASE/Cassandra.
• Perform Unix/Linux shell scripting and related tasks.
• Explore other solutions like Kudu/Impala or Delta Lake.

Skills

Kafka

Spark Streaming

Python/Scala (PySpark/Scala-Spark)

Databricks

Azure/AWS

NoSQL databases

Unix/Linux scripting

Tools

Databricks UI

Delta Lake

Unity Catalog

Kinesis/MSK

Lambda

Glue

TIBCO EMS

IBM MQ Series

Job description

Aptonet is seeking an experienced data engineering professional to design and implement high-velocity data streaming solutions using Apache Kafka and Spark Streaming. You will develop real-time processing with Spark Structured Streaming, troubleshoot Spark workloads, and work with Python/Scala on Databricks in a cloud environment.

Responsibilities include building ingestion pipelines, deploying data platforms on Azure or AWS, managing messaging systems like Kafka/MSK, and handling data formats