Big Data Developer

PradeepIT Consulting Services Pvt Ltd; Bengaluru; On-site; INR 600,000 - 1,000,000; Full time

Job summary

A leading consulting firm in India is looking for a skilled developer to design and implement scalable big data solutions using Hadoop technologies like HBase and Spark. The successful candidate will optimize applications for performance, develop data processing pipelines, and stay current with emerging technologies. Experience with real-time data processing is essential, along with a solid understanding of the Hadoop ecosystem.

Qualifications

• Experience in designing and implementing scalable Hadoop-based solutions.
• Proven ability to optimize Spark applications for performance.
• Familiarity with real-time data processing.

Responsibilities

• Design scalable big data solutions using Hadoop technologies.
• Develop data processing pipelines using Spark.
• Optimize applications for better resource utilization.

Skills

Hadoop ecosystem technologies

Spark

HBase

Hive

Data modeling

Tools

Spark MLlib

Kudu

Job description

Responsibilities

1. Design, develop, and implement highly scalable and distributed big data solutions using Hadoop ecosystem technologies such as HBase, Hive, Kudu, and Spark.
2. Architect HBase schemas and data models to accommodate evolving business requirements and ensure optimal performance for data storage and retrieval operations.
3. Develop complex Hive queries and data processing pipelines to transform raw data into structured formats suitable for analysis and reporting.
4. Implement data ingestion pipelines using Spark Streaming and Spark SQL for real-time processing of streaming data sources, ensuring high throughput and low latency.
5. Optimize Spark applications for performance and resource utilization, including tuning RDD transformations, optimizing data partitioning strategies, and leveraging in-memory caching.
6. Utilize advanced features of Spark MLlib for machine learning tasks such as classification, regression, clustering, and collaborative filtering.
7. Design and deploy Kudu tables for fast analytical queries and real-time analytics, leveraging Kudu's unique combination of fast analytics and fast data ingestion.
8. Collaborate with data scientists to integrate machine learning models into Spark workflows and productionize them for real-time predictions and analytics.
9. Troubleshoot performance bottlenecks, data quality issues, and system failures in big data applications and infrastructure, and implement solutions to address them.
10. Stay abreast of emerging technologies and best practices in big data processing and analytics, and evaluate their potential impact on our architecture and solutions.