Data Engineer

Tekskills; Dadri; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tekskills is seeking a seasoned Big Data Engineer in India to join our data team. The role requires extensive hands-on experience with Python and PySpark, building Spark Dataframe-based applications, and optimizing Spark workloads handling huge data volumes.

Proficiency in Git and AWS cloud services (EMR, Athena, Glue, Lambda, S3) will be highly valued. Familiarity with data warehousing concepts and columnar formats is a plus.

Qualifications

• 5+ years of IT experience with hands-on Big Data technologies.
• Hands-on experience in Python and PySpark is mandatory.
• Experience building PySpark applications using Spark Dataframes in Python.
• Hands-on in version control with Git and AWS data/compute services.

Responsibilities

• Build PySpark applications using Spark Dataframes in Python.
• Optimize Spark jobs that process large volumes of data.
• Work with AWS services like EMR, Athena, Glue, Lambda, and S3.

Skills

Python

PySpark

Spark Dataframes

Big Data

AWS

Tools

Git

AWS EMR

AWS Athena

AWS Glue

Amazon S3

Lambda

Job description

Job Description:

5+ years of overall IT experience, which includes hands on experience in Big Data technologies.

• Mandatory - Hands on experience in Python and PySpark.


• Build pySpark applications using Spark Dataframes in Python.


• Worked on optimizing spark jobs that processes huge volumes of data.


• Hands on experience in version control tools like Git.


• Worked on Amazons Analytics services like Amazon EMR, Amazon Athena, AWS Glue.


• Worked on Amazons Compute services like Amazon Lambda, Amazon EC2 and Amazon Storage service like S3 and few other services like SNS.


• Good to have knowledge of data warehousing concepts dimensions, facts, schemas-snowflake, star etc.


• Have worked with columnar storage formats- Parquet,Avro,ORC etc. Well versed with compression techniques Snappy, Gzip.


• Good to have knowledge of AWS databases (atleast one) Aurora, RDS, Redshift, ElastiCache, DynamoDB.


• Bigdata, AWS, python, Pyspark, AWS services - IAM, lambda, EMR, glue