Senior Data Engineer

YO It Consulting; Chennai; On-site; INR 1,000,000 - 1,400,000; Full time

Job summary

YO It Consulting is seeking a Senior Data Engineer in Chennai to design and develop data engineering solutions, create and maintain data models, and build ETL pipelines.

The ideal candidate will have 6 to 8 years of experience, with strong SQL, Databricks, and PySpark skills. Relevant qualifications include a Bachelor's degree in Computer Science or a related field, along with experience in AWS and big data technologies. Strong analytical and programming skills are essential.

Qualifications

• 6 to 8 years of experience in data engineering.
• Strong programming skills in languages like Python, Java, or Scala.
• Experience with version control tools like Git, CodeCommit.

Responsibilities

• Design and develop data engineering solutions.
• Create and maintain data models.
• Build Extract Transform Load (ETL) pipelines.

Skills

SQL

Databricks

PySpark

Python

Education

Bachelor's degree in Computer Science, Information Systems, or related field

Tools

AWS

Big Data technologies

Job description

Overview

Title : Senior Data Engineer

Experience : 6 to 8 years

Location : Chennai

Must Have :

• SQL
• Databricks
• PySpark
• Python

Role And Responsibilities

The Data Engineer will be responsible for designing and developing data engineering solutions, creating and maintaining data models, building Extract Transform Load (ETL) pipelines, implementing data warehousing, and performing data analytics on large datasets.

• Data Engineering and Data Modeling skills
• Experience with Extract Transform Load (ETL) pipelines
• Expertise in implementing data warehousing solutions
• Analytical skills for performing data analytics on large datasets
• Strong programming skills in languages like Python, Java, or Scala
• Experience with Big Data and cloud-based technologies like Hadoop, Spark, and AWS
• Understanding of data governance and security principles
• Bachelor"s degree in Computer Science, Information Systems, or related field
• Strong communication skills and ability to work independently and collaboratively

Primary Capabilities

SQL Expertise

• Proficiency in writing complex SQL queries.
• Experience with database design and optimization.
• Ability to perform data manipulation and analysis.

Databricks Experience

• Hands-on experience with Databricks platform.
• Familiarity with Databricks" integration with various data sources and sinks.
• Experience in building and optimizing big data pipelines using Databricks.

PySpark Proficiency

• Strong skills in PySpark for large-scale data processing.
• Understanding of PySpark"s architecture and its application in data processing and transformation tasks.
• Ability to write efficient, scalable, and reusable code in PySpark.

Python Programming

• Strong programming skills in Python.
• Experience with Python libraries relevant to data engineering (e.g., Pandas, NumPy).
• Ability to write clean, maintainable, and efficient code in Python.

AWS Experience

• Knowledge of AWS services relevant to data engineering (e.g., S3, Lambda, CodeCommit, etc.,).
• Knowledge in building and maintaining data infrastructures on AWS.
• Understanding of cloud computing concepts and how they apply to data engineering tasks.

Analytics Skills

• Ability to derive insights from large datasets.
• Knowledge of data visualization tools and techniques.
• Proficiency in statistical analysis and using data to inform decisions.

Logical Thinking and Problem-Solving

• Strong logical reasoning and analytical skills.
• Proven ability to troubleshoot and solve complex problems.
• Capacity to break down and simplify complex problems, developing practical, efficient solutions.

Other Desirable Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or a related field.
• Good communication skills and the ability to work collaboratively in a team.
• Experience with version control tools like Git, CodeCommit.
• Familiarity with Agile methodologies and DevOps practices.
• Certification in relevant technologies (e.g., Databricks, AWS Certified Data Analytics) is a plus