Sr. Data Engineer

Orangebitsindia; Chennai District; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading data solutions company is seeking a Senior Data Engineer to transform raw data into useful data systems. This role requires strong analytical skills and proficiency in Python, SQL, and ETL processes. The successful candidate will build and maintain data pipelines and participate in system design meetings. Knowledge of cloud technologies like Azure or AWS and familiarity with tools such as Pyspark or Kafka will be beneficial. Excellent communication skills are also necessary for this role.

Qualifications

• Strong analytical skills required.
• Experience in building data pipelines.
• Familiarity with machine learning methods.

Responsibilities

• Transform raw data into useful data systems.
• Participate in system design meetings.
• Build data pipelines for the data science team.

Skills

Analytical skills

Python

SQL

ETL process

Pyspark

Cloud technologies

Shell Scripting

Kubernetes

Tools

Airflow

Hadoop

SQL Queries

Apache Kafka

Job description

We are looking for senior data engineers to join our team. You will use various methods to transform raw data into useful data systems. For example, you’ll create algorithms and conduct statistical analysis. Overall, you’ll strive for efficiency by aligning data systems with business goals. To succeed in this position, you should have strong analytical skills and the ability to combine data from different sources. Data engineer skills also include familiarity with several programming languages and knowledge of machine learning methods.

Job Requirements

• Participate in the customer’s system design meetings and collect the functional/technicalrequirements.
• Build up data pipelines for consumption by the data science team.
• Skillful in ETL process and tools.
• Clear understanding and experience with Python and PySpark or Spark and SCALA, withHIVE, Airflow, Impala, and Hadoop and RDBMS architecture.
• Experience in writing Python programs and SQL queries.
• Experience in SQL Query tuning.
• Experienced in Shell Scripting (Unix/Linux).
• Build and maintain data pipelines in Spark/Pyspark with SQL and Python or SCALA.
• Knowledge of Cloud (Azure/AWS/GCP, etc..) technologies is additional.
• Good to have knowledge of Kubernetes, CI/CD concepts, Apache Kafka
• Suggest and implement best practices in data integration.
• Guide the QA team in defining system integration tests as needed.
• Split the planned deliverables into tasks and assign them to the team.
• Needs to Maintain/Deploy the ETL code and follow the Agile methodology
• Needs to work on optimization wherever applicable.
• Good oral, written and presentation skills.