DQ Tech Lead

5 Star Recruitment; Bengaluru; Hybrid; INR 3,500,000 - 5,200,000; Full time

Benefits offered by this job

Hybrid work (3 days WFO)

Job summary

5 Star Recruitment seeks an experienced Senior Data Engineer to lead the design, development and deployment of scalable ETL pipelines on cloud platforms. You will work closely with the business owner, product owner and cross-functional teams to gather requirements and translate them into data tooling solutions.

The role requires strong expertise in PySpark/Python, Databricks, AWS cloud, data warehouse concepts, and experience with data quality, testing and validation.

Qualifications

• > Must have experience in building and maintaining reliable and scalable ETL pipelines on big data/Cloud platform
• > Proficiency in scripting languages in PySpark, Python
• > Experience in AWS cloud & DevOps and Databricks
• > Strong data warehouse concepts, Data Lake, Lake House concepts
• > Testing and validation for data transformations and data verification
• > Ability to drive requirements with stakeholders and propose solutions
• > Comfortable mentoring junior engineers in the team (Senior Data Engineers)

Responsibilities

• > Gather technical requirements with the business owner/PO and cross-functional teams
• > Design, develop and deploy data tools supporting operations and product use cases
• > Build and maintain reliable, scalable ETL pipelines on big data/Cloud platforms
• > Collaborate with teams on data infrastructure including relational DBs, Hadoop, and Spark
• > Solve problems in big data technologies and prototype improvements to data architecture
• > Oversee activities of junior data engineering teams
• > Ensure alignment with business vision and objectives

Skills

ETL pipelines

Big data

PySpark

Python

AWS cloud

Databricks

SQL

Hadoop

Spark

Data warehousing

Data Lake

Lake House

Data quality

Testing

Mentoring

Agile / Scrum

Education

16 years education

Tools

Informatica ETL

Ataccama

Databricks

Job description

Works closely with the business owner / Product owner / Cross functional Team and gathering technical requirements

Experience in building and maintaining reliable and scalable ETL pipeline on big data & / or Cloud

platform through the collection, storage, processing, and transformation of large datasets

Collaborate with other teams to design, develop and deploy data tools that support both operationsand product use cases

Work with the team in solving problems in big data technologies and prototype solutions to improveour data processing architecture

Must Have:

Ensuring proper execution of tasks and alignment with business vision and objectives

Oversee activities of the junior data engineering teams

Works closely with the business owner / Product owner / Cross functional Team and gatheringtechnical requirements

Experience in building and maintaining reliable and scalable ETL pipeline on big data/Cloud platformthrough the collection, storage, processing, and transformation of large data-sets

Experience working with varied forms of data infrastructure inclusive of relational databases such asSQL, Hadoop, Spark

Proficiency in scripting languages in PySpark, python

Experience in AWS cloud & DevOps

Experience in Databricks

Must have strong experience in data warehouse concepts, understanding Data Lake, Lake HouseConcepts, any new Big Data eco system.

Experience in testing and validation in order to support the accuracy of data transformations anddata verification

Should be able to independently drive the requirements, have solutioning mindset facing businessstakeholders

Soft Skills:

Excellent Verbal and Communication skills needed.

Should be a excellent team player.

Good knowledge on Agile principles and experience working in scrum teams using Jira

Should be comfortable to mentor junior engineers in the team (For Senior Data Engineers)

Should be able to operate in an ambiguous environment with minimal guidance.

Must-Haves
Experience in Ataccama is mandatory
Experience working with Informatica ETL
16 years full time education
Must have exp into data quality(usually the candidate frow from a data engineer to data quality)
Experience in Databricks, Database design/data modelling
Must have strong experience in data warehouse concepts, understanding Data Lake, Lake House
Mode of work: Hybrid (3 days WFO)
Joining time / Notice Period: Immediate to 30 days