Data Engineer

Orangebitsindia; Chennai District; On-site; INR 1,000,000 - 2,100,000; Full time

Job summary

Orangebitsindia is seeking a Senior Data Engineer to transform raw data into robust systems and drive efficiency by aligning data with business goals.

You will design pipelines, build ETL processes, and work with Python, PySpark, Spark, and Scala, along with Hive, Airflow, Hadoop, and cloud services. Strong analytical skills and cross-source data integration knowledge are essential.

Qualifications

• Participate in customer system design meetings and collect functional/technical requirements.

Responsibilities

• Build data pipelines for consumption by the data science team.
• ETL development and tooling.
• Write Python and SQL queries.
• Tune SQL queries for performance.
• Shell scripting (Unix/Linux) may be involved.
• Develop data pipelines in Spark/PySpark with SQL and Python or Scala.
• Knowledge of Cloud technologies (Azure/AWS/GCP) is a plus.
• Knowledge of Kubernetes and CI/CD concepts is beneficial.
• Support data integration best practices and guide QA on tests.
• Break down deliverables into tasks and assign to the team.
• Maintain/deploy ETL code and follow Agile methods.
• Focus on optimization wherever applicable.
• Communicate effectively in both written and spoken form.

Skills

Python

SQL

PySpark

Spark

Scala

Hive

Airflow

Hadoop

RDBMS

Kubernetes

Kafka

Tools

PySpark

Spark

Hadoop

Airflow

Kafka

Kubernetes

CI/CD

Azure

AWS

GCP

Job description

We are looking for senior data engineers to join our team. You will use various methods to transformraw data into useful data systems. For example, you’ll create algorithms and conduct statisticalanalysis. Overall, you’ll strive for efficiency by aligning data systems with business goals.To succeed in this position, you should have strong analytical skills and the ability to combine datafrom different sources. Data engineer skills also include familiarity with several programminglanguages and knowledge of machine learning methods.

Job Requirements

• Participate in the customer’s system design meetings and collect the functional/technicalrequirements.
• Build up data pipelines for consumption by the data science team.
• Skillful in ETL process and tools.
• Clear understanding and experience with Python and PySpark or Spark and SCALA, withHIVE, Airflow, Impala, and Hadoop and RDBMS architecture.
• Experience in writing Python programs and SQL queries.
• Experience in SQL Query tuning.
• Experienced in Shell Scripting (Unix/Linux).
• Build and maintain data pipelines in Spark/Pyspark with SQL and Python or SCALA.
• Knowledge of Cloud (Azure/AWS/GCP, etc..) technologies is additional.
• Good to have knowledge of Kubernetes, CI/CD concepts, Apache Kafka
• Suggest and implement best practices in data integration.
• Guide the QA team in defining system integration tests as needed.
• Split the planned deliverables into tasks and assign them to the team.
• Needs to Maintain/Deploy the ETL code and follow the Agile methodology
• Needs to work on optimization wherever applicable.
• Good oral, written and presentation skills.