Data Platform Engineer Real-Time Pipelines with Spark/Kafka

Creative Solutions Services, LLC; Cupertino (CA); On-site; USD 83,000 - 103,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401k Program

AD&D Insurance

Employee Assistance

Job summary

NTT DATA is looking for a Data Engineer to join our Cupertino team, focusing on a cloud-native data platform with real-time and batch processing using Spark, Kafka, Flink, and Scala/Java. You will design, build, and maintain scalable data pipelines, connectors, and orchestration on Kubernetes and Azure, driving platform modernization and CI/CD practices.

The role requires deep distributed systems knowledge, strong SQL skills, and hands-on Azure experience, with a focus on enterprise-scale data

Qualifications

• 8+ years of overall Software Engineering or Data Engineering experience.
• 5+ years of hands-on experience with Apache Spark for large-scale batch and streaming data processing.
• 5+ years of hands-on experience with Apache Kafka including event-driven architectures and real-time data streaming solutions.
• Minimum 3+ years of hands-on experience with Apache Flink for stream processing and real-time analytics workloads.
• Minimum 5+ years of experience developing applications using Java and/or Scala.
• Minimum 5+ years of experience writing complex SQL queries and optimizing database performance.
• Minimum 3+ years of experience with Kubernetes and containerized application deployment.
• Minimum 3+ years of experience designing and implementing solutions on Microsoft Azure Cloud.
• Minimum 3+ years of experience building and supporting distributed data platforms using Cassandra, YugabyteDB, PostgreSQL, or similar databases.
• Minimum 3+ years of experience building event-driven architectures and streaming applications.
• Minimum 2+ years of experience implementing CI/CD pipelines, source control, and deployment automation using GitLab, GitHub, or similar tools.
• Minimum 2+ years of experience with cloud-native deployment patterns, containerization, and platform automation.
• Demonstrated experience in performance tuning, troubleshooting, and supporting mission-critical data platforms.

Responsibilities

• Design, develop, and support scalable real-time and batch data pipelines using Apache Spark, Apache Flink, Apache Kafka, and Airflow.
• Build and enhance metadata-driven self-service data integration platforms and reusable connectors.
• Develop and maintain source and target connectors for relational databases, file systems, Kafka, Cassandra, YugabyteDB, and other enterprise data stores.
• Design, deploy, and operate Kubernetes-based streaming and batch processing platforms.
• Lead cloud migration initiatives from on-premise environments to Microsoft Azure.
• Drive performance tuning, scalability optimization, reliability improvements, and operational excellence across large-scale data workloads.
• Develop cloud-native solutions supporting enterprise data movement and processing.
• Collaborate with product owners, architects, cloud engineering teams, and business stakeholders to deliver enterprise-scale data solutions.
• Contribute to platform modernization, automation, CI/CD implementation, and engineering best practices.
• Support troubleshooting, production stability, and continuous improvement initiatives for critical data platforms.

Skills

Spark

Kafka

Flink

Scala

Java

SQL

Kubernetes

Azure

Cassandra

YugabyteDB

PostgreSQL

Airflow

CI/CD

Git

Distributed systems

Platform engineering

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent work experience

Job description

NTT DATA is looking for a Data Engineer to join our Cupertino team, focusing on a cloud-native data platform with real-time and batch processing using Spark, Kafka, Flink, and Scala/Java. You will design, build, and maintain scalable data pipelines, connectors, and orchestration on Kubernetes and Azure, driving platform modernization and CI/CD practices.

The role requires deep distributed systems knowledge, strong SQL skills, and hands-on Azure experience, with a focus on enterprise-scale data