Data Engineer - SQL/PySpark

Forward Eye Technologies; Dadri; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Forward Eye Technologies is seeking a seasoned Data Engineer to design and optimize data processing pipelines. You will write medium-complexity SQL queries, develop PySpark-based ETL jobs, and collaborate to translate data requirements into scalable solutions.

Responsibilities include building scalable data pipelines, implementing cloud-based storage and processing across AWS/Azure/GCP, and ensuring reliability through monitoring and CI/CD integration.

Qualifications

• 5+ years of experience in data engineering.
• Bachelor's degree in CS/IT or related field.
• Hands-on SQL, PySpark, and ETL experience.
• Experience with cloud data platforms (AWS/Azure/GCP).
• Familiarity with CI/CD practices for data projects.

Responsibilities

• Write medium-complexity SQL queries to extract, transform, and load data efficiently.
• Develop and maintain data processing scripts using PySpark or Spark with Scala.
• Design and implement ETL/data engineering pipelines to support BI and analytics needs.
• Work with cloud platforms (AWS, Azure, GCP) to build and manage data solutions.
• Identify, diagnose, and resolve data-related issues; monitor pipelines for reliability.
• Implement and maintain CI/CD processes for data engineering projects.

Skills

SQL queries

PySpark / Spark

ETL design

CI/CD for data projects

Cloud platforms

Data modeling

Problem-solving

Communication

Education

Bachelor's degree in Computer Science/IT

Tools

Airflow

Kafka

AWS EMR / Databricks

Snowflake

Hive

Job description

Key Responsibilities :

SQL Development :

• - Write medium-complexity SQL queries to extract, transform, and load data efficiently.
• - Optimize SQL queries for performance and scalability.
• - Collaborate with team members to understand data requirements and translate them into actionable SQL scripts.

PySpark/Spark Development :

• - Develop and maintain data processing scripts using PySpark or Spark with Scala.
• - Implement data transformation and ETL processes using Spark to handle large-scale data.
• - Optimize Spark jobs for performance and troubleshoot any issues that arise during processing.

Data Engineering Pipeline Design :

• - Design and implement ETL/data engineering pipelines to support business intelligence and analytics needs.
• - Ensure data pipelines are scalable, reliable, and maintainable.
• - Collaborate with the team to understand data flow requirements and implement appropriate solutions.

Cloud Technology Implementation :

• - Work with various cloud platforms (AWS, Azure, GCP) to build and manage data solutions.
• - Implement cloud-based data storage and processing solutions, ensuring security and compliance.
• - Utilize cloud services for data integration, orchestration, and processing.

Issue Resolution and Debugging :

• - Identify, diagnose, and resolve data-related issues and bugs independently.
• - Perform root cause analysis and implement fixes for any data discrepancies or processing failures.
• - Monitor data pipelines and systems to ensure continuous availability and reliability.

CI/CD Integration :

• - Implement and maintain basic to intermediate level CI/CD processes for data engineering projects.
• - Collaborate with DevOps teams to ensure smooth deployment and integration of data solutions.
• - Automate testing and deployment processes to improve development efficiency.

Desired Skills :

Data Modeling :

• - Experience in designing data models to support analytical and business requirements.

Airflow :

• - Experience with Apache Airflow for workflow scheduling and orchestration.

Kafka :

• - Knowledge of Kafka for real-time data streaming and integration.

EMR/Databricks :

• - Experience with AWS EMR or Databricks for big data processing and analytics.

Snowflake :

• - Familiarity with Snowflake for cloud-based data warehousing solutions.

Hive :

• - Understanding of Hive for querying and managing large datasets.

Qualifications :

• - Experience : 5+ years of experience in data engineering or related fields.

Technical Skills :

• - Strong proficiency in SQL and writing medium-complexity queries.
• - Hands-on experience with PySpark or Spark with Scala.
• - Understanding of ETL processes and data engineering pipeline design.
• - Basic to intermediate knowledge of CI/CD processes.
• - Experience with cloud technologies (AWS, Azure, GCP).

Problem-Solving :

• - Ability to troubleshoot and resolve data issues independently.

Communication :

• - Strong verbal and written communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders.

Education :

• - Bachelor's degree in Computer Science, Information Technology, or related field.