Data Engineer AWS Pyspark

Zensar Technologies; Hyderabad; On-site; INR 1,700,000 - 2,100,000; Full time

Job summary

Zensar Technologies in Hyderabad is seeking a Data Engineer with strong AWS and PySpark experience to design and implement scalable data pipelines. You will build PySpark applications, optimize Spark jobs, and work with data in Parquet formats on AWS data services.

You should have 6+ years of relevant Big Data experience, hands-on Python and PySpark, and familiarity with Git version control. Knowledge of data warehousing concepts and AWS databases is a plus.

Qualifications

• 6+ years of relevant experience in Big Data technologies.
• Hands on experience in Python and PySpark.
• Experience building PySpark applications using Spark Dataframes in Python.
• Optimizing Spark jobs handling large data volumes.
• Hands on experience with Git or similar VCS.
• Experience with AWS analytics services (EMR, Athena, Glue).
• Experience with AWS compute and storage (Lambda, EC2, S3, SNS).
• Knowledge of data warehousing concepts (dimensions, facts, schemas).
• Familiarity with Parquet and compression (Snappy, Gzip).
• Knowledge of AWS databases (Aurora, RDS, Redshift, ElastiCache, DynamoDB)

Responsibilities

• Design and implement scalable data pipelines in AWS cloud.
• Develop PySpark applications and Spark Dataframes in Python.
• Optimize Spark jobs for large volumes of data.
• Collaborate with teams to integrate data sources and warehouses.
• Maintain code with Git and implement data quality checks.

Skills

Python

PySpark

Spark Dataframes

Big Data

Cloud Computing

Git

AWS Services

Aurora

Redshift

S3

EC2

Lambda

EMR

Athena

Glue

DynamoDB

Parquet

Snappy

Gzip

Data Warehousing

Snowflake schema

Tools

Git

S3

EC2

Lambda

EMR

Athena

Glue

Redshift

DynamoDB

Aurora

RDS

ElastiCache

Job description

Data Engineer (AWS + pySpark)

• Having 6+ yrs years of relevant experience, which includes hands on experience in Big Data technologies.
• Mandatory - Hands on experience in Python and PySpark.
• Build pySpark applications using Spark Dataframes in Python.
• Worked on optimizing spark jobs that processes huge volumes of data.
• Hands on experience in version control tools like Git.
• Worked on Amazon’s Analytics services like Amazon EMR, Amazon Athena, AWS Glue.
• Worked on Amazon’s Compute services like Amazon Lambda, Amazon EC2 and Amazon’s Storage service like S3 and few other services like SNS.
• Good to have knowledge of datawarehousing concepts – dimensions, facts, schemas- snowflake, star etc.
• Have worked with columnar storage formats - Parquet etc. Well versed with compression techniques – Snappy, Gzip.
• Good to have knowledge of AWS databases (atleast one) Aurora, RDS, Redshift, ElastiCache, DynamoDB.am