Sr Data Engineering

EXL; Bengaluru; On-site; USD 908 - 1,704; Full time

Job summary

An established industry player seeks a skilled Data Engineer to design and maintain ETL processes. In this exciting role, you will collaborate with stakeholders to understand data requirements, create data models, and manage data pipelines that ensure quality and consistency. The ideal candidate will have extensive experience in data engineering, particularly with Python, Pyspark, and Azure technologies. Join a dynamic team where your contributions will drive impactful data solutions and enhance business intelligence. If you're passionate about data and eager to make a difference, this opportunity is perfect for you.

Qualifications

• 7+ years of experience in Data Engineering with strong Python and Pyspark skills.
• Experience in Azure cloud computing and developing ETL processes.

Responsibilities

• Design, develop, and maintain complex ETL processes based on stakeholder requirements.
• Create and manage data pipelines ensuring quality and consistency.

Skills

Python

Pyspark

SQL

Hadoop

Hive

Azure

Databricks

Greenplum

Data Quality Testing

Problem-Solving

Education

Degree in Data Science

Degree in Statistics

Degree in Computer Science

Tools

Azure Data Factory

Hue

Apache Oozie

Job description

Get AI-powered advice on this job and more exclusive features.

Responsibilities:

• Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
• Create the data integration and data diagram documentation.
• Lead the data validation, UAT and regression test for new data asset creation.
• Create and maintain data models, including schema design and optimization.
• Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.

Qualifications and Skills:

• Strong knowledge on Python and Pyspark.
• Ability to write Pyspark scripts for developing data workflows.
• Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum.
• Ability to write SQL to query metadata and tables from different data management systems such as Oracle, Hive, Databricks and Greenplum.
• Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.
• Experience using Hue and running Hive SQL queries, scheduling Apache Oozie jobs to automate data workflows.
• Good experience communicating with stakeholders and collaborating effectively with the business team for data testing.
• Strong problem-solving and troubleshooting skills.
• Ability to establish comprehensive data quality test cases, procedures and implement automated data validation processes.
• Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.
• 7+ years of experience in Data Engineering.
• Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.
• Experience in Azure cloud computing platform, including developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.
• Strong communication, problem-solving and analytical skills with the ability to manage time and multi-task with attention to detail and accuracy.

Seniority Level:

Mid-Senior level

Employment Type:

Full-time

Job Function:

Consulting

Industries:

Business Consulting and Services