Data Engineer-ADF

Prudent Technologies and Consulting, Inc.; Hyderabad; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A technology consulting firm in Hyderabad seeks a Data Engineer to define, design, and build data pipelines and analytics models. You will work with various data technologies, enhance existing processes, and collaborate with stakeholders to optimize data infrastructure. The ideal candidate has strong skills in Python, SQL, and data visualization tools such as PowerBI and Tableau, ensuring actionable insights into business performance metrics. This role promises a dynamic environment with a focus on data excellence.

Qualifications

• Experience with data pipeline architecture, ELT & ETL processes.
• Build and deploy analytical models and tools for insights.
• Experience working with stakeholders on data infrastructure issues.

Responsibilities

• Define, design, and build data pipelines and models.
• Implement internal process improvements and automated processes.
• Troubleshoot and maintain data pipelines, prepare RCA.

Skills

Python

Spark

Snowflake

Databricks

Hadoop

Hive

Sqoop

Oozie

SQL

PowerBI

Tableau

Looker

R

Job description

• Define, Design, and Build an optimal data pipeline architecture to collect data from a variety of sources, cleanse, and organize data in SQL & NoSQL destinations (ELT & ETL Processes).
• Define and Build business use case-specific data models that can be consumed by Data Scientists and Data Analysts to conduct discovery and drive business insights and patterns.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
• Build and deploy analytical models and tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
• Work with stakeholders including the Executive, Product, Data, and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Define, Design, and Build Executive dashboards and reports catalogs to serve decision-making and insight generation needs.
• Provide inputs to help keep data separated and secure across data centers – on-prem and private and public cloud environments.
• Create data tools for analytics and data science team members that assist them in building and optimizing our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Implement scheduled data load process and maintain and manage the data pipelines.
• Troubleshoot, investigate, and fix failed data pipelines and prepare RCA.

Experience with a mix of the following Data Engineering Technologies

• Python, Spark, Snowflake, Databricks, Hadoop (CDH), Hive, Sqoop, oozie

Experience with a mix of the following Data Analytics and Visualization toolsets

• SQL, PowerBI, Tableau, Looker, Python, R