Data Engineer-Spark, Scala, Spark Streaming, Big data Hadoop, hive, Kafka, Cassandra - Developer

Tata Consultancy Services; Austin (TX); On-site; USD 70,000 - 140,000; Full time

Job summary

A leading company seeks a Data Engineer proficient in Spark and Scala to develop scalable solutions. Responsibilities include managing data pipelines, collaborating with stakeholders, and optimizing processes. Ideal candidates should have strong communication skills and familiarity with Agile methodologies.

Qualifications

• Strong experience in Spark, Scala, SQL, Kafka, Cassandra, Spark Streaming.
• Good to have AWS experience.
• Knowledge of Python and Unix shell scripting.

Responsibilities

• Developing scalable solutions using Spark, Scala, and Big Data Hadoop.
• Managing system integrations and optimizing data pipelines.
• Gathering business requirements and delivering user training.

Skills

Spark

Scala

SQL

Kafka

Cassandra

Spark Streaming

Python

Unix shell scripting

Communication

Problem Solving

Job description

Join to apply for the Data Engineer - Spark, Scala, Spark Streaming, Big Data Hadoop, Hive, Kafka, Cassandra - Developer role at Tata Consultancy Services.

Responsibilities include:

1. Developing and implementing scalable solutions using technologies like Spark, Scala, and Big Data Hadoop.
2. Managing system integrations, optimizing data pipelines, and providing technical support.
3. Gathering business requirements, designing functional specifications, streamlining processes, conducting system testing, and delivering user training.
4. Collaborating with stakeholders and technical teams to align business goals with technical solutions.

Skills & Qualifications:

• Strong experience in Spark, Scala, SQL (basic and advanced), Kafka, Cassandra, Spark Streaming.
• Good to have AWS experience.
• Knowledge of Python and Unix shell scripting.
• Expertise in business process mapping, requirements gathering, and creating functional specifications.
• Excellent communication and problem-solving skills.
• Familiarity with Agile methodologies.

Salary Range: $70,000 - $140,000 per year.