Aws Data Engineer

Paltech Consulting; Hyderabad; Hybrid; INR 1,400,000 - 2,100,000; Full time

Job description

Job Description AWS Data Engineer (Redshift)

Location: Hyderabad | Work From Office

Experience: 48 Years

Notice Period: Immediate to 30 Days preferred

We are looking for an experienced AWS Data Engineer with strong hands-on experience in Amazon Redshift, AWS Glue, S3, SQL and PySpark to design and build scalable data pipelines and cloud data warehouse solutions.

Key Responsibilities

• Design, develop and maintain ETL/ELT pipelines using AWS services.
• Build and optimize data pipelines using AWS Glue, S3 and PySpark.
• Develop and maintain Amazon Redshift data warehouse solutions.
• Load and transform data from S3 and various source systems into Redshift.
• Write complex and optimized SQL queries, stored procedures and data transformations.
• Design appropriate Redshift distribution keys and sort keys.
• Perform Redshift query and workload optimization, including identifying data skew, inefficient joins and unnecessary data scans.
• Implement incremental loads, CDC and data-quality checks.
• Work with Redshift Spectrum for querying data directly from S3 where appropriate.
• Implement data partitioning and file-format optimization, preferably using Parquet.
• Develop scalable PySpark transformations for large datasets.
• Implement monitoring, logging and failure/retry mechanisms for production pipelines.
• Work with IAM, CloudWatch, Lambda/EventBridge and other AWS services as required.
• Participate in CI/CD and deployment automation for data pipelines.
• Troubleshoot production data pipeline and Redshift performance issues.

Required Skills

• 48 years of Data Engineering experience.
• Strong hands-on experience with Amazon Redshift.
• Strong SQL skills.
• Hands-on experience with AWS Glue.
• Strong experience with Amazon S3.
• Experience with PySpark / Apache Spark.
• Strong understanding of ETL/ELT architecture.
• Experience with incremental loading and data-quality validation.
• Good understanding of cloud data warehouse concepts.

Redshift-specific expertise

• COPY command and S3-to-Redshift loading.
• DISTKEY / SORTKEY selection.
• Distribution styles.
• Query optimization.
• Data skew.
• VACUUM / ANALYZE concepts.
• Redshift Spectrum.
• Workload management.
• Large-volume data loading and optimization.

Good to Have

• AWS Lambda
• EventBridge
• CloudWatch
• Athena
• EMR
• IAM
• Airflow / MWAA
• dbt
• Terraform
• CI/CD
• Python

Ideal Candidate

We are looking for someone who has genuine hands-on Redshift experience, not just Redshift listed in the resume. The candidate should be able to explain a complete architecture such as:

Source Systems S3 AWS Glue/PySpark Redshift BI/Analytics

and confidently troubleshoot Redshift performance, data loading, distribution, sorting and SQL optimization.