Senior Data Engineer

atsMantra; Bengaluru; On-site; INR 1,000,000 - 2,000,000; Full time

Job summary

atsMantra is seeking a seasoned Data Engineer in Bengaluru to design and implement large-scale data pipelines using Spark, PySpark, and AWS EMR. You will optimize Hadoop/Hive-based data stores, write robust Python/Scala code, and ensure efficient data retrieval from S3 and Elasticsearch.

Knowledge of Airflow and Autosys for orchestrating ETL jobs is essential. The role requires 5+ years in data engineering, strong SQL skills, Unix scripting, and experience with API development and integration.

Qualifications

• 5+ years of data engineering experience
• Degree in Computer Science, Engineering, or related field

Skills

Apache Spark

PySpark

AWS EMR

Hadoop

Hive

SQL

Python

Scala

Unix

Databricks

Education

Bachelor's or Master's degree in Computer Science or related field

Tools

Airflow

Autosys

Git

SVN

Elasticsearch

Job description

• Education: Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Experience: 5+ years of experience in data engineering or a similar role.

Technical Skills:

• Proficiency in Apache SPARK, pySpark, and AWS EMR.
• Strong experience with Apache Hadoop, Hive, and SQL (Postgres-SQL).
• Advanced programming skills in Python and Scala.
• Solid understanding of UNIX systems and shell scripting.
• Must have Databricks knowledge and experience

Functional understanding of API

• Experience with data storage and retrieval from S3 and Elasticsearch.
• Knowledge of ETL tools like Airflow and Autosys.
• Familiarity with version control systems (Git, SVN).
• Basic understanding of HTML for web development tasks.
• Strong SQL skills for data querying and manipulation.
• Experience with API development and integration.