Senior Hadoop & Spark Engineer - 10+ yrs

Tata Consultancy Services; Charlotte (NC); On-site; USD 95,000 - 115,000; Full time

Benefits offered by this job

Annual incentive

Medical coverage

Parental leaves

Insurance options

Certification & training

Job summary

Tata Consultancy Services in Charlotte, NC is seeking an experienced Hadoop Spark Developer with 10+ years of Big Data expertise, including Java, Spark, Hadoop, and Hive. Responsibilities include designing, developing, and maintaining large-scale data processing solutions and collaborating with stakeholders.

The role emphasizes building ETL/ELT pipelines, real-time streaming with Kafka, data quality frameworks, and code reviews.

Qualifications

• 10+ years of experience in Hadoop/Spark ecosystem.
• Proficient in Spark, Java, Hive, SQL.
• Unix, Kafka, Python skills.

Responsibilities

• Design, develop, and maintain scalable Big Data solutions using Hadoop and Spark.
• Build and optimize ETL/ELT pipelines using Java, Spark, Hive, and Python.
• Process and analyze large datasets in distributed environments.
• Develop high-performance Spark jobs and optimize existing workloads.
• Create and manage Hive tables, partitions, views, and complex queries.
• Implement data quality, data validation, and reconciliation frameworks.
• Perform code reviews and ensure adherence to coding standards and best practices.
• Utilize Microsoft Copilot to accelerate development, automate code generation, troubleshooting, documentation, and testing activities.
• Strong experience building both batch and real-time streaming applications with Kafka.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and DevOps teams.
• Troubleshoot production issues and perform root cause analysis.
• Design data ingestion frameworks for structured, semi-structured, and unstructured data.
• Participate in Agile ceremonies including sprint planning, estimation, and retrospectives.
• Mentor junior developers and provide technical leadership.

Skills

Spark

Java

Hive

SQL

Unix

Kafka

Python

Education

Bachelor's degree

Job description

Tata Consultancy Services in Charlotte, NC is seeking an experienced Hadoop Spark Developer with 10+ years of Big Data expertise, including Java, Spark, Hadoop, and Hive. Responsibilities include designing, developing, and maintaining large-scale data processing solutions and collaborating with stakeholders.

The role emphasizes building ETL/ELT pipelines, real-time streaming with Kafka, data quality frameworks, and code reviews.