Data Engineer - SQL/PySpark

Forward Eye Technologies; Pune District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Forward Eye Technologies is seeking a data engineer to design and optimize data pipelines. The role focuses on SQL development and PySpark-based processing, building scalable ETL with cloud platforms such as AWS, Azure, or GCP.

The ideal candidate will model data, work with data scientists, and ensure reliability, performance, and maintainability of data infrastructure across BI and analytics workloads.

Qualifications

• 5+ years of experience in data engineering or related field.
• Strong proficiency in SQL and writing medium-complexity queries.
• Hands-on experience with PySpark or Spark with Scala.
• Understanding of ETL processes and data engineering pipeline design.
• Basic to intermediate knowledge of CI/CD processes.
• Experience with cloud technologies (AWS, Azure, GCP).

Responsibilities

• Write medium-complexity SQL queries to extract, transform, and load data.
• Optimize SQL queries for performance and scalability.
• Develop and maintain data processing scripts using PySpark or Spark with Scala.
• Design and implement ETL/data engineering pipelines to support BI and analytics.
• Work with cloud platforms (AWS/Azure/GCP) to build data solutions.
• Identify, diagnose, and resolve data-related issues and bugs.

Skills

SQL

PySpark

Data Modeling

Airflow

Kafka

CI/CD

Cloud

Education

Bachelor's degree in CS/IT

Tools

Airflow

Kafka

EMR/Databricks

Snowflake

Hive

Job description

We are seeking a skilled Data Engineer with strong experience in SQL, PySpark, and cloud technologies to join our dynamic team. The ideal candidate will have a solid background in designing and implementing data engineering pipelines, with a focus on performance, scalability, and reliability. You will work closely with other data engineers, data scientists, and stakeholders to develop, maintain, and optimize data infrastructure.

Key Responsibilities

SQL Development

• Write medium-complexity SQL queries to extract, transform, and load data efficiently.
• Optimize SQL queries for performance and scalability.
• Collaborate with team members to understand data requirements and translate them into actionable SQL scripts.

PySpark/Spark Development

• Develop and maintain data processing scripts using PySpark or Spark with Scala.
• Implement data transformation and ETL processes using Spark to handle large-scale data.
• Optimize Spark jobs for performance and troubleshoot any issues that arise during processing.

Data Engineering Pipeline Design

• Design and implement ETL/data engineering pipelines to support business intelligence and analytics needs.
• Ensure data pipelines are scalable, reliable, and maintainable.
• Collaborate with the team to understand data flow requirements and implement appropriate solutions.

Cloud Technology Implementation

• Work with various cloud platforms (AWS, Azure, GCP) to build and manage data solutions.
• Implement cloud-based data storage and processing solutions, ensuring security and compliance.
• Utilize cloud services for data integration, orchestration, and processing.

Issue Resolution and Debugging

• Identify, diagnose, and resolve data-related issues and bugs independently.
• Perform root cause analysis and implement fixes for any data discrepancies or processing failures.
• Monitor data pipelines and systems to ensure continuous availability and reliability.

CI/CD Integration

• Implement and maintain basic to intermediate level CI/CD processes for data engineering projects.
• Collaborate with DevOps teams to ensure smooth deployment and integration of data solutions.
• Automate testing and deployment processes to improve development efficiency.

Desired Skills

Data Modeling

• Experience in designing data models to support analytical and business requirements.

Airflow

• Experience with Apache Airflow for workflow scheduling and orchestration.

Kafka

• Knowledge of Kafka for real-time data streaming and integration.

EMR/Databricks

• Experience with AWS EMR or Databricks for big data processing and analytics.

Snowflake

• Familiarity with Snowflake for cloud-based data warehousing solutions.

Hive

• Understanding of Hive for querying and managing large datasets.

Qualifications

Experience : 5+ years of experience in data engineering or related fields.

Technical Skills

• Strong proficiency in SQL and writing medium-complexity queries.
• Hands-on experience with PySpark or Spark with Scala.
• Understanding of ETL processes and data engineering pipeline design.
• Basic to intermediate knowledge of CI/CD processes.
• Experience with cloud technologies (AWS, Azure, GCP).

Problem-Solving

• Ability to troubleshoot and resolve data issues independently.

Communication

• Strong verbal and written communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders.

Education

• Bachelor's degree in Computer Science, Information Technology, or related field.