GCP Data Engineer

Tech Holding; Pune District; On-site; INR 800,000 - 1,200,000; Full time

Job summary

Tech Holding is seeking a Data Engineer to design, develop, and maintain data pipelines that handle large volumes of data across cloud environments. You will contribute to scalable, reliable data solutions supporting business initiatives.

The role involves collaborating with cross-functional teams, implementing best practices, and working across the SDLC to deliver high-quality data processing applications in a fast-paced consulting setting.

Qualifications

• Big data concepts and GCP stack: Cloud Composer, BigQuery, Dataproc, Dataflow, Cloud Logging, Cloud Functions, GCS.
• CI/CD experience and strong Linux and shell scripting for automation.
• Experience with scheduling tools.
• Analytical/technical skills to design, develop, and optimize data pipelines.
• Excellent verbal and written communication skills.
• Experience with Agile methodologies.

Responsibilities

• Focuses on the processing, analyzing, monitoring, and extracting data from the cloud servers
• Deliver reliable, scalable, and efficient application managing all phases of SDLC, including requirements gathering, analysis, design, development, testing, deployment, integration, and documentation
• Understand the requirements (both functional and non-functional) with inputs from business analysts and participate in creating high level estimate
• Work with cross-functional teams to develop solutions that is accurate and of good quality
• Design the system architecture, data model, information flow, input, and output configurations
• Develop tables, jobs, views, scheduler process, etc as per business rules
• Perform QA testing, support UAT activities and provide production support, as well as handle change requests and incidents within the agreed SLA
• Identify and implement process improvements to drive innovation
• Participate in retrospective meeting and work on continuous improvement of delivery quality and timeline with automation, scalability, and defect reduction
• Provide the necessary guidance/support to the rest of the team to achieve the team deliverables
• Stay updated with the latest trends and developments in the data engineering field, such as cloud big data technologies

Skills

Big data concepts

GCP

HDFS

SQL

Hive

PySpark

Shell scripting

CI/CD

Linux

Agile

Tools

Cloud Composer

BigQuery

Dataproc

Dataflow

Cloud Logging

Cloud Functions

GCS

Job description

About us: Working at Tech Holding isnt just a job, its an opportunity to be a part of something bigger We are a full-service consulting firm that was founded on the premise of delivering predictable outcomes and high-quality solutions to our clients Our founders and team members have industry experience and have held senior positions in a wide variety of companies- from emerging startups to large Fortune 50 firms- and we have taken our combined experiences and developed a unique approach that is supported by the principles of deep expertise, integrity, transparency, and dependability

• We are looking for talented Data Engineer to join our team
• As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines and working with large volumes of data to support our business initiatives

Responsibilities

• Focuses on the processing, analyzing, monitoring, and extracting data from the cloud servers
• Deliver reliable, scalable, and efficient application managing all phases of SDLC, including requirements gathering, analysis, design, development, testing, deployment, integration, and documentation
• Understand the requirements (both functional and non-functional) with inputs from business analysts and participate in creating high level estimate
• Work with cross-functional teams to develop solutions that is accurate and of good quality
• Design the system architecture, data model, information flow, input, and output configurations
• Develop tables, jobs, views, scheduler process, etc as per business rules
• Perform QA testing, support UAT activities and provide production support, as well as handle change requests and incidents within the agreed SLA
• Identify and implement process improvements to drive innovation
• Participate in retrospective meeting and work on continuous improvement of delivery quality and timeline with automation, scalability, and defect reduction
• Provide the necessary guidance/support to the rest of the team to achieve the team deliverables
• Stay updated with the latest trends and developments in the data engineering field, such as cloud big data technologies

Requirements

• Skill setsBig data concepts, GCP (Cloud Composer, BigQuery, Dataproc, Dataflow, Cloud Logging, Cloud Functions, GCS), HDFS, SQL, Hive, PySpark, Shell script
• Experience with CI/CD methodologies and tools, including strong Linux and shell scripting skills for automation
• Knowledge on any of the scheduling tools
• Good team player with strong analytical/technical skills to analyze data pipeline and develop codes as per business requirement
• Expertise in designing, developing, and optimizing data pipelines, ensuring robust and scalable solutions
• Excellent verbal and written communication skills
• Ability to work under pressure and meet deadlines, while maintaining a positive attitude
• Ability to work independently and to carry out the tasks to completion quickly while maintaining quality
• Experience with Agile methodology

Tech Holding is proud to be an Equal Opportunity Employer and is committed to fostering a diverse and inclusive workplace We welcome applicants from all backgrounds and experiences, and we consider qualified applicants without regard to race, color, religion, gender, sexual orientation, gender identity, national origin, disability, veteran status, or any other legally protected characteristic If you require accommodation in the application process, please contact our HR

Show more Show less