Aws Data Engineer

Shrewd Techlink Services; Hyderabad, Pune District, Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Shrewd Techlink Services in Hyderabad is seeking an experienced AWS Data Engineer to design, build, and maintain scalable data pipelines, data lakes, and data warehouse solutions on AWS. You will work with Databricks, PySpark, and Spark to process large datasets and enable analytics across the business.

You will optimize ETL/ELT workflows, ensure data quality, governance, and security while collaborating with data scientists and stakeholders to translate requirements into robust data solutions.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• Strong SQL knowledge and data warehousing concepts.
• Hands-on experience with Databricks, Spark, PySpark in cloud data pipelines.
• Experience with Git version control and CI/CD practices.
• Experience with Delta Lake and performance tuning.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using PySpark and Databricks.
• Build and optimize data ingestion frameworks from various structured and unstructured data sources.
• Develop and manage data lakes and data warehouse solutions on AWS.
• Implement data transformation, cleansing, validation, and enrichment processes.
• Work with large datasets and optimize Spark jobs for performance and scalability.
• Integrate data from APIs, databases, streaming platforms, and third‑party systems.
• Collaborate with Data Scientists, Analysts, and Business stakeholders to understand data requirements.
• Monitor, troubleshoot, and enhance data pipeline performance and reliability.
• Implement data quality, governance, security, and compliance best practices.
• Participate in code reviews, architecture discussions, and technical design sessions.

Skills

AWS Cloud Services

Databricks

Apache Spark

PySpark

Python

SQL

Delta Lake

Git

CI/CD

Data Modeling

Education

Bachelor's or Master's degree in CS/IT/Engineering

Tools

Spark SQL

Delta Lake

Job description

Job Summary

We are seeking an experienced AWS Data Engineer with strong expertise in Databricks, PySpark, and cloud-based data engineering solutions. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, data lakes, and data warehouse solutions on AWS. The role requires hands‑on experience in processing large‑scale datasets, optimizing data workflows, and supporting analytics and business intelligence initiatives.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using PySpark and Databricks.
• Build and optimize data ingestion frameworks from various structured and unstructured data sources.
• Develop and manage data lakes and data warehouse solutions on AWS.
• Implement data transformation, cleansing, validation, and enrichment processes.
• Work with large datasets and optimize Spark jobs for performance and scalability.
• Integrate data from APIs, databases, streaming platforms, and third‑party systems.
• Collaborate with Data Scientists, Analysts, and Business stakeholders to understand data requirements.
• Monitor, troubleshoot, and enhance data pipeline performance and reliability.
• Implement data quality, governance, security, and compliance best practices.
• Participate in code reviews, architecture discussions, and technical design sessions.

Required Skills

• Strong experience in AWS Cloud Services (S3, EMR, Glue, Lambda, IAM, Redshift, Athena, CloudWatch).
• Hands‑on experience with Databricks and Apache Spark.
• Strong programming skills in PySpark and Python.
• Experience in building ETL/ELT pipelines and data integration solutions.
• Strong knowledge of SQL and relational databases.
• Experience with Delta Lake, Spark SQL, and performance tuning.
• Understanding of data modeling concepts and data warehousing principles.
• Experience with version control systems such as Git.
• Knowledge of CI/CD practices and deployment automation.

Preferred Skills

• Experience with Apache Kafka or other streaming technologies.
• Knowledge of Airflow, AWS Step Functions, or workflow orchestration tools.
• Experience with Snowflake or Redshift.
• Familiarity with Terraform or Infrastructure as Code (IaC).
• Exposure to machine learning data pipelines and MLOps.
• AWS Certification is a plus.

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Key Competencies

• Strong analytical and problem‑solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work independently and in a collaborative environment.
• Strong focus on performance optimization and data quality.