Data Engineer - SQL/PySpark

Forward Eye Technologies; Dadri; Hybrid; INR 1,400,000 - 2,000,000; Full time

Job summary

Forward Eye Technologies is seeking a skilled Data Engineer to design and optimize data pipelines using SQL, PySpark, and cloud technologies. You will ensure performance, scalability, and reliability while collaborating with data scientists and stakeholders.

You will develop ETL processes, implement Spark jobs, and build data architectures on AWS/Azure/GCP, focusing on robust data models, data quality, and maintainable solutions for analytics and BI.

Qualifications

• 5+ years of experience in data engineering or related fields.
• Strong proficiency in SQL and writing medium-complexity queries.
• Hands-on experience with PySpark or Spark with Scala.
• Understanding of ETL processes and data engineering pipeline design.
• Basic to intermediate knowledge of CI/CD processes.
• Experience with cloud technologies (AWS, Azure, GCP).

Responsibilities

• SQL development to extract, transform, and load data efficiently.
• Develop and maintain data processing scripts using PySpark or Spark.
• Design and implement ETL/data pipelines for BI and analytics.
• Work with cloud platforms to build data solutions and ensure security.
• Identify and resolve data issues; monitor pipelines for reliability.
• Implement and maintain CI/CD processes for data projects.

Skills

SQL

PySpark

Cloud

Data Modeling

Airflow

Kafka

EMR/Databricks

Snowflake

Hive

CI/CD

Education

Bachelor's degree in CS/IT or related field

Tools

SQL

PySpark

Spark with Scala

AWS/Azure/GCP

Job description

We are seeking a skilled Data Engineer with strong experience in SQL, PySpark, and cloud technologies to join our dynamic team. The ideal candidate will have a solid background in designing and implementing data engineering pipelines, with a focus on performance, scalability, and reliability. You will work closely with other data engineers, data scientists, and stakeholders to develop, maintain, and optimize data infrastructure.

Key Responsibilities :

• SQL Development
  • Write medium-complexity SQL queries to extract, transform, and load data efficiently.
  • Optimize SQL queries for performance and scalability.
  • Collaborate with team members to understand data requirements and translate them into actionable SQL scripts.
• PySpark/Spark Development
  • Develop and maintain data processing scripts using PySpark or Spark with Scala.
  • Implement data transformation and ETL processes using Spark to handle large-scale data.
  • Optimize Spark jobs for performance and troubleshoot any issues that arise during processing.
• Data Engineering Pipeline Design
  • Design and implement ETL/data engineering pipelines to support business intelligence and analytics needs.
  • Ensure data pipelines are scalable, reliable, and maintainable.
  • Collaborate with the team to understand data flow requirements and implement appropriate solutions.
• Cloud Technology Implementation
  • Work with various cloud platforms (AWS, Azure, GCP) to build and manage data solutions.
  • Implement cloud-based data storage and processing solutions, ensuring security and compliance.
  • Utilize cloud services for data integration, orchestration, and processing.
• Issue Resolution and Debugging
  • Identify, diagnose, and resolve data-related issues and bugs independently.
  • Perform root cause analysis and implement fixes for any data discrepancies or processing failures.
  • Monitor data pipelines and systems to ensure continuous availability and reliability.
• CI/CD Integration
  • Implement and maintain basic to intermediate level CI/CD processes for data engineering projects.
  • Collaborate with DevOps teams to ensure smooth deployment and integration of data solutions.
  • Automate testing and deployment processes to improve development efficiency.

Desired Skills :

• Data Modeling
  • Experience in designing data models to support analytical and business requirements.
• Airflow
  • Experience with Apache Airflow for workflow scheduling and orchestration.
• Kafka
  • Knowledge of Kafka for real-time data streaming and integration.
• EMR/Databricks
  • Experience with AWS EMR or Databricks for big data processing and analytics.
• Snowflake
  • Familiarity with Snowflake for cloud-based data warehousing solutions.
• Hive
  • Understanding of Hive for querying and managing large datasets.

Qualifications :

Experience : 5+ years of experience in data engineering or related fields.

Technical Skills :

• Strong proficiency in SQL and writing medium-complexity queries.
• Hands-on experience with PySpark or Spark with Scala.
• Understanding of ETL processes and data engineering pipeline design.
• Basic to intermediate knowledge of CI/CD processes.
• Experience with cloud technologies (AWS, Azure, GCP).

Problem-Solving :

Ability to troubleshoot and resolve data issues independently.

Communication :

Strong verbal and written communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders.

Education :

- Bachelor's degree in Computer Science, Information Technology, or related field.

Location : - Anywhere india ,Multiple Locations