Senior Big Data Engineer

Cortex Consultants LLC; Bengaluru Urban; Hybrid; INR 5,000,000 - 7,000,000; Full time

Benefits offered by this job

Health Insurance

Retirement Benefits

Paid Time Off

Flexible Benefits Program

Job summary

Cortex Consultants LLC is seeking a Senior Big Data Engineer in Bangalore (Nagavara, near Manyata Tech Park). This hybrid role requires 8+ years in data engineering, with hands-on Spark and Python, plus experience on AWS or Azure Databricks. Immediate to 15 days notice preferred.

You will design scalable data systems, optimize pipelines, and mentor an Agile team while collaborating across functional units on large-scale projects.

Qualifications

• 8+ years of experience in Big Data and Data Engineering technologies.
• Expert-level experience with Apache Spark.
• Hands-on programming experience in Python.
• Good knowledge of Hadoop ecosystem (Hadoop v2, HDFS, MapReduce, Sqoop).
• Experience with stream processing technologies such as Spark Streaming or Apache Storm.
• Experience with messaging platforms like Kafka or RabbitMQ.
• Good understanding of Hive and Impala.
• Experience integrating data from SQL Server, Oracle, ERP systems, and other enterprise data sources.
• Strong SQL skills including joins, stored procedures, and relational database concepts.
• Experience with NoSQL databases such as HBase, Cassandra, or MongoDB.
• Strong understanding of ETL frameworks and best practices.
• Experience in Spark performance tuning.

Responsibilities

• Design, develop, and implement scalable Big Data solutions.
• Build high-performance data pipelines using Apache Spark and Python.
• Develop and optimize ETL processes for large-scale data processing.
• Integrate data from multiple sources including RDBMS, ERP systems, and files.
• Optimize Spark jobs for performance and scalability.
• Work with cloud-based data platforms using AWS Databricks or Azure Databricks.
• Lead technical initiatives and mentor team members.
• Collaborate with cross-functional teams following Agile methodologies.

Skills

Apache Spark

Python

Databricks

AWS or Azure

Tools

Hadoop

Hive

Kafka

MongoDB

Job description

Job Opportunity: Senior Big Data Engineer Databricks (AWS/Azure)

Position: Senior Big Data Engineer

Employment Type: Full-Time (FTE)

Experience: 8+ Years

Location: Bangalore (Nagavara, Near Manyata Tech Park)

Work Mode: Hybrid Client Location (2 3 days/week)

Notice Period: Immediate to 15 Days Preferred

Interview Mode: Face-to-Face (All interview rounds will be completed on the same day)

Mandatory Skills

• Apache Spark
• Python
• Databricks (AWS Databricks or Azure Databricks) Mandatory
• AWS or Azure Cloud

Key Responsibilities

• Design, develop, and implement scalable Big Data solutions.
• Build high-performance data pipelines using Apache Spark and Python.
• Develop and optimize ETL processes for large-scale data processing.
• Integrate data from multiple sources including RDBMS, ERP systems, and files.
• Optimize Spark jobs for performance and scalability.
• Work with cloud-based data platforms using AWS Databricks or Azure Databricks.
• Lead technical initiatives and mentor team members.
• Collaborate with cross-functional teams following Agile methodologies.

Required Skills & Experience

• 8+ years of experience in Big Data and Data Engineering technologies.
• Strong understanding of distributed computing concepts.
• Expert-level experience with Apache Spark.
• Hands-on programming experience in Python.
• Good knowledge of Hadoop ecosystem (Hadoop v2, HDFS, MapReduce, Sqoop).
• Experience with stream processing technologies such as Spark Streaming or Apache Storm.
• Experience with messaging platforms like Kafka or RabbitMQ.
• Good understanding of Hive and Impala.
• Experience integrating data from SQL Server, Oracle, ERP systems, and other enterprise data sources.
• Strong SQL skills including joins, stored procedures, and relational database concepts.
• Experience with NoSQL databases such as HBase, Cassandra, or MongoDB.
• Strong understanding of ETL frameworks and best practices.
• Experience in Spark performance tuning.
• Hands-on experience with Azure Databricks or AWS Databricks.Ability to lead technical teams and deliver end-to-end Big Data solutions.
• Experience working in Agile development environments.

What We Offer

• Opportunity to work on large-scale global projects.
• Continuous learning through online training, knowledge-sharing sessions, and external certifications.
• International collaboration and opportunities to contribute on global platforms.
• Sponsored Tech Talks and Hackathons.
• Opportunities for short-term and long-term relocation to global offices.
• Individual career development and growth plans.
• Comprehensive benefits including:
  • Health & Medical Insurance
  • Retirement Benefits
  • Paid Time Off
  • Flexible Benefits Program
• Employee engagement forums including CSR initiatives, photography, sports, painting, and other interest groups.