GCP Cloud- Senior Data Architect- PAN India

Tredence Inc.; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Tredence Inc. in Bengaluru, India, seeks an experienced Big Data Engineer to lead data engineering efforts across collecting, storing, processing, and analyzing large data sets. You will integrate these solutions into client architectures and mentor a team of data engineers, data scientists and software engineers.

You will drive scalable analytics by leveraging Hadoop, Spark, Kafka and related tooling, while shaping data models and pipelines for optimal performance.

Qualifications

• Experience with data warehousing and data modeling.
• Strong knowledge of distributed computing and big data tooling.
• Hands-on with Spark, SQL, Linux and ETL pipelines.
• Experience with streaming systems like Storm or Spark Streaming.
• Familiarity with Lambda Architecture and its trade-offs.

Responsibilities

• Select and integrate Big Data tools and frameworks to deliver capabilities.
• Develop and maintain ETL data pipelines and monitor performance.
• Translate complex requirements into detailed data designs.
• Investigate alternative data storage/processing approaches for efficiency.
• Mentor junior staff through technical training and code reviews.

Skills

Data warehousing

Data modeling

Distributed computing

Streaming systems

Spark

SQL

Linux

ETL techniques

Kafka

Hadoop

HDFS

Hive

Impala

NiFi

DBT

Lambda Architecture

Tools

Hadoop ecosystem

Storm

Spark Streaming

Pig

Hive

Impala

Flume

Kafka

RabbitMQ

DBT

NiFi

Job description

In this rapidly evolving world, Tredence is always looking for new ways to disrupt the status quo, go to market faster, and optimize customer experiences. We are seeking an experienced product engineer to lead the charge on this mission. The ideal candidate will have thorough experience in executing ML and DS solutions and managing and collaborating with a team of data engineers, data scientists and software engineers. They will foster an environment of collaboration and creativity. Most importantly, they will be integral to helping us build better products that people [or businesses] love to use.

About Tredence

Tredence focuses on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science and software engineering. The largest companies across industries are engaging with us and deploying their prediction and optimization solutions at scale. Head quartered in the San Francisco Bay Area, we serve clients in the US, Canada, Europe, and South East Asia.

We are looking for a Big Data Engineer that will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across our clients.

Primary Roles and Responsibilities

• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Develop and maintain data pipelines implementing ETL process, monitoring performance and advising any necessary infrastructure changes
• Translate complex technical and functional requirements into detailed designs
• Investigate and analyze alternative solutions to data storing, processing etc. to ensure most streamlined approaches are implemented
• Serve as a mentor to junior staff by conducting technical training sessions and reviewing project outputs

Skills and Qualifications

• Strong understanding of data warehousing and data modeling techniques.
• Proficient understanding of distributed computing principles - Hadoop v2, MapReduce, HDFS
• Experience with building stream-processing systems, using solutions such as Storm or Spark- Streaming
• Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
• Experience with Spark, SQL, and Linux.
• Knowledge of various ETL techniques and frameworks, such as Flume, Apache NiFi, or DBT.
• Experience with various messaging systems, such as Kafka or RabbitMQ.
• Good understanding of Lambda Architecture, along with its advantages and drawbacks

Experience - 10 to 15 years

Notice Period- 0-15 days