Big Data Developer

Consign Space Solutions Llp; Hyderabad; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

Consign Space Solutions Llp in Hyderabad is seeking a seasoned Big Data Engineer to design, develop, and maintain scalable data pipelines using Hadoop ecosystem technologies. You will build batch and real-time processing solutions, develop ETL/ELT workflows, and collaborate with data scientists and analysts to deliver robust data solutions.

Responsibilities include optimizing Hadoop clusters for performance and reliability, ensuring data quality and governance, and supporting CI/CD automation

Qualifications

• Experience designing and building scalable data pipelines with Hadoop ecosystem.
• Strong knowledge of batch and real-time processing.
• Hands-on ETL/ELT development for large datasets.
• Experience with Spark, Hive, and MapReduce.
• Experience managing Hadoop clusters for performance and reliability.
• Ability to collaborate with data scientists and analysts.
• Focus on data quality, monitoring, and security best practices.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Hadoop ecosystem technologies.
• Build and optimize batch and real-time data processing solutions.
• Develop ETL/ELT workflows for ingesting, transforming, and processing large datasets.
• Work with Spark, Hive, and MapReduce as distributed computing frameworks.
• Manage and optimize Hadoop clusters for performance, scalability, and reliability.
• Integrate data from multiple data sources, structured and unstructured.
• Collaborate with data scientists, analysts, and application teams.
• Monitor data quality, troubleshoot production issues, and optimize systems.
• Implement security and governance for big data environments.
• Support CI/CD, automation, and deployment of big data applications

Skills

Hadoop

Spark

Hive

MapReduce

ETL/ELT

Data pipelines

CI/CD

Data governance

Data quality

Big data security

Tools

Spark

Hive

MapReduce

Job description

Role & responsibilities

• Design, develop, and maintain scalable data pipelines using Hadoop ecosystem technologies.
• Build and optimize batch and real-time data processing solutions.
• Develop ETL/ELT workflows for ingesting, transforming, and processing large datasets.
• Work with distributed computing frameworks such as Spark, Hive, and MapReduce.
• Manage and optimize Hadoop clusters for performance, scalability, and reliability.
• Integrate data from multiple structured and unstructured data sources.
• Collaborate with data scientists, analysts, and application teams to deliver data solutions.
• Monitor data quality, troubleshoot production issues, and optimize system performance.
• Implement security, governance, and best practices for big data environments.
• Support CI/CD, automation, and deployment of big data application

Preferred candidate profile