Big Data Lead

Veriipro; United States; On-site; USD 180,000 - 240,000; Full time

Job summary

Veriipro is seeking a highly experienced Big Data Lead to design, develop, and manage scalable data processing solutions using the Hadoop ecosystem and modern data engineering technologies. The ideal candidate will lead and mentor teams while delivering robust data pipelines for large-scale analytics.

You will leverage Apache Spark, Scala, Python, Hadoop, and cloud platforms (AWS or GCP) to build and optimize systems, ensure reliability, and collaborate with data scientists and engineers to meet

Qualifications

• 10+ years of experience in Big Data / Data Engineering.
• Hands-on with Hadoop ecosystem including Hadoop, Hive, Pig, Oozie.
• Extensive Apache Spark expertise and Spark RDD APIs.
• Proficiency in Scala and Python programming.
• Experience with Core Java, SQL, and Linux environments.
• Hands-on with Kafka or similar streaming frameworks.
• Strong cloud experience on AWS or GCP.

Responsibilities

• Lead design and development of scalable big data platforms and pipelines.
• Develop and optimize batch and real-time processing with Spark.
• Implement distributed data processing using Spark RDDs.
• Build and maintain ETL workflows for large data volumes.
• Utilize Hadoop ecosystem tools (Hadoop, Hive, Pig, Oozie).
• Create streaming data pipelines with Kafka.
• Collaborate with data scientists and engineers to support analytics.
• Ensure performance, scalability, and reliability of data systems.
• Provide mentorship to junior data engineers and guide teams.
• Deploy and manage data infrastructure on AWS or GCP.
• Maintain Linux-based data processing environments.

Skills

Big Data engineering

Spark

Scala

Python

Java

SQL

Linux

Kafka

Tools

Hadoop

Hive

Pig

Oozie

Job description

We are seeking a highly experiencedBig Data Lead to design, develop, and manage scalable data processing solutions using the Hadoop ecosystem and modern data engineering technologies. The ideal candidate will have strong hands-on expertise in Apache Spark, Scala, Python, Hadoop, and cloud platforms (AWS or GCP). This role requires technical leadership in building robust data pipelines, supporting large-scale analytics, and mentoring data engineering teams.

Key Responsibilities

• Lead the design and development of scalable big data platforms and data pipelines.
• Develop and optimize batch and real-time data processing solutions using Apache Spark.
• Work with Spark RDD APIs and implement distributed data processing applications.
• Build and maintain ETL workflows for processing large volumes of structured and unstructured data.
• Utilize Hadoop ecosystem tools such as Hadoop, Hive, Pig, and Oozie.
• Implement streaming data pipelines using technologies like Kafka.
• Collaborate with data scientists, analysts, and engineering teams to support analytics and business intelligence requirements.
• Ensure performance tuning, scalability, and reliability of big data systems.
• Provide technical guidance and mentorship to junior data engineers.
• Work with cloud platforms (AWS or GCP) to deploy and manage data infrastructure.
• Maintain and support Linux-based environments for data processing systems.

Required Skills

• 10+ years of experience in Big Data, Data Engineering, or related roles.
• Strong hands-on experience with Hadoop ecosystem technologies including Hadoop, Hive, Pig, and Oozie.
• Extensive experience with Apache Spark and strong knowledge of Spark RDD APIs.
• Proficiency in Scala and Python programming.
• Experience with Core Java.
• Strong knowledge of SQL and scripting languages.
• Hands-on experience with Kafka or similar streaming frameworks.
• Strong experience working in Linux environments.

Data & Analytics Knowledge

• Strong understanding of Data Warehousing concepts.
• Experience with Data Modeling techniques.
• Familiarity with data visualization and analytics tools such as Tableau or R.

Cloud Experience

• Hands-on experience with AWS or GCP cloud platforms.
• Experience building data engineering workflows and big data solutions in cloud environments.

Preferred Qualifications

• Experience designing real-time streaming architectures.
• Strong experience in performance tuning and optimization for Spark and Hadoop.
• Prior experience leading data engineering teams or projects.