Data Engineer - ETL/PySpark

Forward Eye Technologies; Dadri; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Forward Eye Technologies is seeking a data engineer to design and optimize ETL/ELT pipelines using PySpark or Spark with Scala. You will write advanced SQL for data extraction, build streaming pipelines with Kafka, and orchestrate workflows with Airflow in a cloud environment.

You will collaborate with data architects, work on data lakes and warehousing solutions, and implement CI/CD practices to ensure reliable deployments.

Qualifications

• Write complex SQL queries for data extraction, transformation and analysis.
• Design and optimize SQL queries for performance and maintainability.
• Develop, test and deploy data processing apps using PySpark or Spark with Scala.
• Implement and maintain ETL/ELT data pipelines for robust data processing.

Responsibilities

• Design and implement robust ETL pipelines for ingestion, transformation, and loading.
• Collaborate with data architects and engineers to ensure seamless data flow.
• Leverage cloud platforms (AWS, Azure, Google Cloud) to build scalable data solutions.
• Work with cloud-based data storage (S3, Google Cloud Storage) and compute environments.
• Debug data processing issues and resolve challenges independently as an individual contributor.
• Troubleshoot and optimize existing pipelines to improve reliability and performance.
• Implement and manage CI/CD pipelines for automated testing, deployment, and monitoring.
• Ensure code quality and best practices through version control (Git) and CI/CD tools.

Skills

PySpark

SQL

Spark/Scala

Airflow

Kafka

Cloud platforms

Data modeling

CI/CD

ETL/ELT design

Python

Education

Bachelor's degree in CS/IT/Data Science

Masters preferred

Tools

Hive

EMR

Databricks

Snowflake

S3 / Cloud Storage

Job description

Job Responsibilities :

• Write medium to complex SQL queries to support data extraction, transformation, and analysis.
• Optimize SQL queries for performance and maintainability.PySpark/Spark Development :
• Develop, test, and deploy data processing applications using PySpark or Spark with Scala.
• Implement and maintain ETL/ELT data pipelines, ensuring efficient data processing and integration.ETL/ Data Engineering Pipeline Design :
• Design and implement robust ETL pipelines to support data ingestion, transformation, and loading.
• Collaborate with data architects and engineers to ensure seamless data flow and integration across systems.
• Leverage cloud platforms (AWS, Azure, Google Cloud, etc.) to build scalable and efficient data solutions.
• Work with cloud
• based data storage solutions (e.g., S3, Google Cloud Storage) and compute environments.
• Debug data processing issues and resolve challenges independently as an individual contributor.
• Troubleshoot and optimize existing pipelines to improve reliability and performance.Continuous Integration and Continuous Deployment (CI/CD) :
• Implement and manage CI/CD pipelines for automated testing, deployment, and monitoring.
• Ensure code quality and best practices through version control (e.g., Git) and CI/CD tools.

Desired Skills

Data Modeling

• Design and maintain logical and physical data models to support business requirements.
• Ensure data models are optimized for performance, scalability, and ease of use.Airflow :

Airflow

• Use Apache Airflow to orchestrate and schedule complex data workflows.
• Develop and maintain DAGs for task automation and monitoring.

Kafka

• Utilize Apache Kafka for building real-time data streaming pipelines.
• Integrate Kafka with other systems for event-driven data processing.Big Data Technologies :

Big Data Technologies

• Experience with big data processing frameworks such as Hive, EMR, and Databricks.
• Familiarity with data storage solutions like Snowflake for analytics and reporting.Data Lakes and Data Warehousing :

Data Lakes and Data Warehousing

• Work with data lake solutions and data warehouses for storing and managing large datasets.
• Implement data lake architectures for scalable data ingestion and processing.

Technical Knowledge and Skills Required

• Strong hands-on experience with PySpark or Spark with Scala for big data processing.
• Proficient in writing complex SQL queries and optimizing them for performance.
• Understanding of ETL/ELT design principles and best practices.
• Basic to intermediate knowledge of CI/CD processes and tools.
• Familiarity with cloud platforms (AWS, Azure, Google Cloud) and their data -related services.
• Ability to work independently and resolve issues effectively.

Soft Skills Required

• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Ability to manage multiple tasks and projects efficiently.
• A proactive attitude towards learning new technologies and improving existing skills.

Work Experience

• 5+ years of relevant experience in data engineering, focusing on big data processing and cloud technologies.
• Experience working with data engineering tools and frameworks like Airflow, Kafka, Hive, EMR, Databricks, and Snowflake.Education :

Education

• Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field.
• A Master's degree is a plus.