Data Engineer - Only Immediate Joiner

Tekskills Inc.; Kolkata District; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Tekskills Inc. is seeking a Big Data Engineer with hands-on Python and PySpark experience in Kolkata. You will build PySpark applications, optimize Spark jobs handling large datasets, and work with AWS services such as EMR, Glue, Athena, Lambda, EC2, and S3.

Familiarity with data warehousing concepts and columnar formats is a plus. Responsibilities include developing scalable data pipelines, collaborating via Git, and applying best practices for data storage, querying, and processing in a

Qualifications

• Hands-on Python and PySpark development
• Experience building PySpark apps using Spark Dataframes in Python
• GIT version control experience and collaboration workflows
• Experience with AWS analytics services (EMR, Glue, Athena) and compute/storage (Lambda, EC2, S3)
• Familiarity with data warehousing concepts (dimensions, facts, schemas) like star/snowflake
• Knowledge of columnar formats (Parquet/Avro/ORC) and compression (Snappy/Gzip)
• Knowledge of AWS databases (Aurora, RDS, Redshift, DynamoDB)

Responsibilities

• Develop and optimize PySpark jobs processing large data volumes
• Build and maintain Spark Dataframe pipelines in Python
• Work with AWS services for data storage, processing, and analytics
• Collaborate using Git and CI/CD practices
• Apply data warehousing concepts and columnar formats in projects

Skills

Python

PySpark

Spark Dataframes

Git

AWS services

Tools

Amazon EMR

Amazon Athena

AWS Glue

Amazon Lambda

Amazon EC2

Amazon S3

SNS

Parquet/Avro/ORC

Snappy/Gzip

Aurora/RDS/Redshift

DynamoDB/ElastiCache

Job description

• Mandatory - Hands on experience in Python and PySpark.

• Build pySpark applications using Spark Dataframes in Python.

• Worked on optimizing spark jobs that processes huge volumes of data.

• Hands on experience in version control tools like Git.

• Worked on Amazons Analytics services like Amazon EMR, Amazon Athena, AWS Glue.

• Worked on Amazon's Compute services like Amazon Lambda, Amazon EC2 and Amazon's Storage service like S3 and few other services like SNS.

• Good to have knowledge of datawarehousing concepts - dimensions, facts, schemas- snowflake, star etc.

• Have worked with columnar storage formats- Parquet,Avro,ORC etc. Well versed with compression techniques - Snappy, Gzip.

• Good to have knowledge of AWS databases (atleast one) Aurora, RDS, Redshift, ElastiCache, DynamoDB.

Bigdata, AWS, python, Pyspark, AWS services - IAM, lambda, EMR, glue