Data Engineer - Only Immediate Joiner

Tekskills Inc.; Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tekskills Inc. is seeking a data engineer with hands-on Python and PySpark experience to build and optimize Spark Dataframes-based applications. You will work with AWS services such as EMR, Athena, and Glue, and leverage S3 for storage.

A solid understanding of data warehousing concepts and columnar formats will help you design efficient schemas and compress data effectively. You'll collaborate across teams, maintain code with Git, and contribute to scalable data solutions in a fast-paced

Qualifications

• Hands-on Python and PySpark development experience.
• Experience building PySpark applications using Spark Dataframes in Python.
• Proven ability to optimize Spark jobs processing large data volumes.
• Version control experience with Git.
• Hands-on with AWS analytics services (EMR, Athena, Glue).
• Familiarity with AWS compute and storage services (Lambda, EC2, S3).
• Knowledge of data warehousing concepts and star/snowflake schemas.
• Experience with columnar data formats (Parquet, Avro, ORC) and compression (Snappy, Gzip).
• Knowledge of AWS databases (Aurora, RDS, Redshift, ElastiCache, DynamoDB).

Responsibilities

• Develop PySpark applications using Spark Dataframes in Python.
• Optimize Spark jobs for large-scale data processing.
• Collaborate with teams on AWS data services integration.
• Maintain version control with Git and CI/CD pipelines.
• Work with data warehousing concepts and schema design.

Skills

Python

PySpark

Spark Dataframes

Git

AWS EMR

AWS Athena

AWS Glue

AWS Lambda

AWS EC2

AWS S3

Data Warehousing Concepts

Columnar Formats (Parquet/Avro/ORC)

Compression (Snappy/Gzip)

AWS Databases (Aurora/RDS/Redshift/Elb

Tools

Git

Amazon EMR

Amazon Athena

AWS Glue

Amazon Lambda

Amazon EC2

Amazon S3

Parquet/Avro/ORC

Snowflake

Job description

• Mandatory - Hands on experience in Python and PySpark.

• Build pySpark applications using Spark Dataframes in Python.

• Worked on optimizing spark jobs that processes huge volumes of data.

• Hands on experience in version control tools like Git.

• Worked on Amazons Analytics services like Amazon EMR, Amazon Athena, AWS Glue.

• Worked on Amazon's Compute services like Amazon Lambda, Amazon EC2 and Amazon's Storage service like S3 and few other services like SNS.

• Good to have knowledge of datawarehousing concepts - dimensions, facts, schemas- snowflake, star etc.

• Have worked with columnar storage formats- Parquet,Avro,ORC etc. Well versed with compression techniques - Snappy, Gzip.

• Good to have knowledge of AWS databases (atleast one) Aurora, RDS, Redshift, ElastiCache, DynamoDB.

Bigdata, AWS, python, Pyspark, AWS services - IAM, lambda, EMR, glue