Big Data Engineer

Alike Thoughts; Hyderabad, Chennai District; Hybrid; INR 1,200,000 - 2,600,000; Full time

Job summary

Alike Thoughts in Hyderabad, India is seeking a skilled Big Data Engineer to design and implement scalable data pipelines using Hadoop and Spark. You will optimize PySpark jobs, integrate with Hive, Sqoop, Oozie and Kafka, and collaborate with cross-functional teams to deliver data-driven insights for business goals.

The role focuses on maintaining data quality, governance, and secure, reliable data infrastructure while staying current with emerging big data technologies and best practices.

Qualifications

• Experience with Hadoop and Spark ecosystems.
• Proficient in Python for data processing.
• Ability to design scalable data workflows.
• Knowledge of data governance and quality.

Responsibilities

• Design and deploy scalable big data solutions with Hadoop and Spark.
• Implement Spark SQL and PySpark jobs for performance.
• Monitor Hadoop clusters for stability and capacity planning.
• Collaborate with data engineers to deliver actionable insights.
• Conduct code reviews and mentor junior team members.
• Participate in Agile ceremonies and release cycles.
• Document technical specs and best practices in big data projects.
• Ensure security and compliance in big data environments.

Skills

Big Data

Spark

PySpark

Python

Tools

Hive

Sqoop

Oozie

Kafka

Job description

Job Description

• Utilize the Big Data Hadoop Ecosystem to efficiently process and analyze large datasets
• Develop and optimize data workflows leveraging Hadoop Spark and Spark SQL for largescale data analytics
• Develop Python applications and scripts to automate data processing and integration within the Hadoop ecosystem
• Collaborate with cross functional teams to understand data requirements and deliver optimized solutions aligned with business goals
• Ensure data quality integrity and governance across distributed systems
• Integrate various data ingestion and transformation tools such as HDFS Hive Sqoop Oozie and Kafka to support data processing needs
• Stay updated with emerging technologies and best practices in Hadoop Spark Python and big data frameworks
• Optimize system performance and troubleshoot issues related to big data infrastructure to ensure high availability and reliability

Roles and Responsibilities

• Design develop and deploy scalable big data solutions using Hadoop and Spark ecosystems
• Implement and optimize Spark SQL and PySpark jobs to enhance performance and resource utilization
• Monitor and maintain Hadoop clusters for stability capacity planning and performance tuning
• Collaborate with data engineers analysts and stakeholders to deliver actionable insights and high-quality data solutions
• Conduct code reviews provide technical guidance and mentor junior team members
• Participate actively in Agile ceremonies including sprint planning code reviews and release cycles
• Document technical specifications processes and best practices related to big data projects
• Ensure security and compliance standards are met within big data environments