Data Engineer - SQL/PySpark

Forward Eye Technologies; Pune District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Forward Eye Technologies seeks an experienced Data Engineer to design, build, and maintain data pipelines and ETL processes. You will develop SQL and Spark-based data solutions, optimize performance, and collaborate with cross-functional teams on data requirements.

Candidate should have 5+ years in data engineering, strong cloud experience (AWS/Azure/GCP), and a solid foundation in CI/CD practices. Join a team focused on scalable, reliable data infrastructure.

Qualifications

• 5+ years of experience in data engineering or related fields.
• Strong SQL skills with medium-complexity queries.
• Hands-on PySpark or Spark with Scala experience.
• Understanding of ETL processes and data pipelines.
• Basic to intermediate knowledge of CI/CD processes.
• Experience with cloud technologies (AWS, Azure, GCP).
• Strong problem-solving and communication skills.

Responsibilities

• Write medium-complexity SQL queries to extract, transform, and load data efficiently.
• Optimize SQL queries for performance and scalability.
• Collaborate with team members to understand data requirements and translate them into actionable SQL scripts.
• Develop and maintain data processing scripts using PySpark or Spark with Scala.
• Implement data transformation and ETL processes using Spark to handle large-scale data.
• Optimize Spark jobs for performance and troubleshoot any issues that arise during processing.
• Design and implement ETL/data engineering pipelines to support business intelligence and analytics needs.
• Ensure data pipelines are scalable, reliable, and maintainable.
• Collaborate with the team to understand data flow requirements and implement appropriate solutions.
• Work with various cloud platforms (AWS, Azure, GCP) to build and manage data solutions.
• Implement cloud-based data storage and processing solutions, ensuring security and compliance.
• Utilize cloud services for data integration, orchestration, and processing.
• Identify, diagnose, and resolve data-related issues and bugs independently.
• Perform root cause analysis and implement fixes for any data discrepancies or processing failures.
• Monitor data pipelines and systems to ensure continuous availability and reliability.
• Implement and maintain basic to intermediate level CI/CD processes for data engineering projects.
• Collaborate with DevOps teams to ensure smooth deployment and integration of data solutions.
• Automate testing and deployment processes to improve development efficiency.

Skills

SQL proficiency

Spark / PySpark

ETL design

CI/CD basics

Communication

Education

Bachelor's degree in CS/IT

Tools

Airflow

Kafka

AWS EMR/Databricks

Snowflake

Hive

Databricks

Job description

Key Responsibilities :

SQL Development :

• - Write medium-complexity SQL queries to extract, transform, and load data efficiently.
• - Optimize SQL queries for performance and scalability.
• - Collaborate with team members to understand data requirements and translate them into actionable SQL scripts.

PySpark/Spark Development :

• - Develop and maintain data processing scripts using PySpark or Spark with Scala.
• - Implement data transformation and ETL processes using Spark to handle large-scale data.
• - Optimize Spark jobs for performance and troubleshoot any issues that arise during processing.

Data Engineering Pipeline Design :

• - Design and implement ETL/data engineering pipelines to support business intelligence and analytics needs.
• - Ensure data pipelines are scalable, reliable, and maintainable.
• - Collaborate with the team to understand data flow requirements and implement appropriate solutions.

Cloud Technology Implementation :

• - Work with various cloud platforms (AWS, Azure, GCP) to build and manage data solutions.
• - Implement cloud-based data storage and processing solutions, ensuring security and compliance.
• - Utilize cloud services for data integration, orchestration, and processing.

Issue Resolution and Debugging :

• - Identify, diagnose, and resolve data-related issues and bugs independently.
• - Perform root cause analysis and implement fixes for any data discrepancies or processing failures.
• - Monitor data pipelines and systems to ensure continuous availability and reliability.

CI/CD Integration :

• - Implement and maintain basic to intermediate level CI/CD processes for data engineering projects.
• - Collaborate with DevOps teams to ensure smooth deployment and integration of data solutions.
• - Automate testing and deployment processes to improve development efficiency.

Desired Skills :

Data Modeling :

• - Experience in designing data models to support analytical and business requirements.

Airflow :

• - Experience with Apache Airflow for workflow scheduling and orchestration.

Kafka :

• - Knowledge of Kafka for real-time data streaming and integration.

EMR/Databricks :

• - Experience with AWS EMR or Databricks for big data processing and analytics.

Snowflake :

• - Familiarity with Snowflake for cloud-based data warehousing solutions.

Hive :

• - Understanding of Hive for querying and managing large datasets.

Qualifications :

• - Experience : 5+ years of experience in data engineering or related fields.

Technical Skills :

• - Strong proficiency in SQL and writing medium-complexity queries.
• - Hands-on experience with PySpark or Spark with Scala.
• - Understanding of ETL processes and data engineering pipeline design.
• - Basic to intermediate knowledge of CI/CD processes.
• - Experience with cloud technologies (AWS, Azure, GCP).

Problem-Solving :

• - Ability to troubleshoot and resolve data issues independently.

Communication :

• - Strong verbal and written communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders.

Education :

• - Bachelor's degree in Computer Science, Information Technology, or related field.