Data Engineer - ETL

Forward Eye Technologies; Pune District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Forward Eye Technologies in India is seeking a skilled Data Engineer to design, develop, and optimize large-scale data pipelines on AWS using PySpark and SQL.

You will collaborate with data science teams, implement data quality and governance, and contribute to real-time and batch processing with Airflow-driven workflows.

This role offers exposure to AWS data services such as S3, Glue, Redshift, EMR, and Lambda, with a focus on scalable, secure analytics infrastructure.

Qualifications

• 5+ years of data engineering experience focused on PySpark, AWS, and SQL.
• Hands-on experience with ETL and data pipeline development.
• Experience with data orchestration and monitoring (Airflow or similar).
• Strong understanding of cloud data infrastructure and security best practices.
• Proficient in writing optimized SQL queries for large datasets.

Responsibilities

• Design, build, and optimize ETL pipelines using PySpark on AWS.
• Collaborate with data engineering and data science teams to deliver scalable data solutions.
• Develop and manage AWS data services (S3, Glue, Redshift, EMR, Lambda).
• Write and optimize SQL queries for data extraction and transformation.
• Implement data quality, governance, and security practices.
• Automate workflows and maintain CI/CD pipelines for data jobs.
• Troubleshoot data pipelines and support analytics initiatives.

Skills

PySpark

AWS

SQL

Airflow

Git

CI/CD

Tools

S3

Glue

Redshift

EMR

Lambda

Job description

We are looking for a skilled Data Engineer with expertise in PySpark, AWS, and SQL to support data processing and analytical initiatives. This role involves working closely with data engineering and data science teams to build, maintain, and optimize large-scale data pipelines and integrations on AWS. The ideal candidate will be proficient in ETL processes using PySpark and SQL, with a deep understanding of cloud data infrastructure, specifically within the AWS ecosystem.

Key Responsibilities :

Data Pipeline Development :

• Design, build, and optimize ETL pipelines using PySpark for data ingestion, transformation, and storage on AWS.
• Collaborate with stakeholders to understand data requirements, translating them into scalable data solutions.

Cloud Infrastructure Management :

• Develop and manage AWS services such as S3, Glue, Lambda, EMR, Redshift, and RDS for data processing and storage.
• Implement data workflows that handle both batch and real-time processing needs, ensuring low-latency and efficient data access.

Database Management :

• Write and optimize complex SQL queries for data extraction and transformation from AWS RDS, Redshift, and other SQL-based databases.
• Leverage indexing, partitioning, and caching techniques to improve query performance on large datasets.

Data Quality and Governance :

• Ensure data accuracy, completeness, and consistency throughout the data lifecycle.
• Implement best practices for data quality, governance, and security using AWS-native tools and third-party solutions.

Automation and Optimization :

• Automate repetitive tasks and optimize workflows, ensuring efficient and resilient data processing.
• Use CI/CD pipelines and version control (e.g., Git) to deploy and manage data workflows.

Troubleshooting and Support :

• Identify and resolve issues within the data pipeline, providing solutions for data access and query performance.
• Support the team in data analytics and data science initiatives by preparing data in an accessible and structured format.

Required Skills and Experience :

• 5+ years of data engineering experience with a focus on PySpark, AWS, and SQL.
• Strong proficiency in PySpark for ETL and data transformation tasks.
• Hands-on experience with AWS data services (S3, Glue, Redshift, EMR, Lambda).
• Expertise in SQL and relational database management, with experience optimizing complex queries.
• Experience in data pipeline orchestration and monitoring (Airflow or similar).
• Knowledge of data partitioning, indexing, and caching techniques to enhance performance.
• Familiarity with CI/CD principles and tools, including version control systems like Git.

Nice to Have :

• Experience with AWS Redshift Spectrum and Athena for querying data on S3.
• Knowledge of streaming data processes and tools like Kinesis or Kafka.
• Background in big data technologies, such as Hadoop or Apache Spark.
• Experience with data lake architectures and building data marts for analytics.

Location : - Anywhere india ,Multiple Locations