Data Engineer - ETL/PySpark

Forward Eye Technologies; Pune District; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Forward Eye Technologies is seeking a senior Data Engineer to develop and optimize data pipelines, write complex SQL for analytics, and implement scalable ETL/ELT processes using PySpark/Spark with Scala. The role emphasizes cloud-native architectures and CI/CD for reliable delivery.

The ideal candidate will have 5+ years in data engineering, hands-on experience with Airflow, Kafka, and Snowflake, and be able to work across data lakes and warehouses to enable real-time analytics.

Qualifications

• 5+ years of relevant experience in data engineering.
• Proficient with data modeling and cloud-based data solutions.
• Experience with Airflow, Kafka, Hive, EMR, Databricks, Snowflake.

Responsibilities

• Write medium to complex SQL queries for data extraction and analysis.
• Develop, test, and deploy data processing apps using PySpark/Spark with Scala.
• Design and maintain ETL/ELT data pipelines for scalable data processing.
• Leverage cloud platforms (AWS, Azure, Google Cloud) for scalable data solutions.
• Implement and manage CI/CD pipelines for automated testing and deployment.

Skills

Big data

Analytical thinking

Problem-solving

Independent work

Education

Bachelor's degree

Master's degree

Tools

PySpark/Spark with Scala

Airflow

Kafka

Snowflake

Hive

Databricks

EMR

Job description

Job Responsibilities :

SQL Development :

• - Write medium to complex SQL queries to support data extraction, transformation, and analysis.
• - Optimize SQL queries for performance and maintainability.PySpark/Spark Development :
• - Develop, test, and deploy data processing applications using PySpark or Spark with Scala.
• - Implement and maintain ETL/ELT data pipelines, ensuring efficient data processing and integration.ETL/ Data Engineering Pipeline Design :

Cloud Technology Utilization :

• - Leverage cloud platforms (AWS, Azure, Google Cloud, etc.) to build scalable and efficient data solutions.
• - Work with cloud
• -based data storage solutions (e.g., S3, Google Cloud Storage) and compute environments.

Debugging and Issue Resolution :

• - Debug data processing issues and resolve challenges independently as an individual contributor.
• - Troubleshoot and optimize existing pipelines to improve reliability and performance.Continuous Integration and Continuous Deployment (CI/CD) :

Continuous Integration and Continuous Deployment (CI/CD) :

• - Implement and manage CI/CD pipelines for automated testing, deployment, and monitoring.
• - Ensure code quality and best practices through version control (e.g., Git) and CI/CD tools.

Desired Skills :

Data Modeling :

• - Design and maintain logical and physical data models to support business requirements.
• - Ensure data models are optimized for performance, scalability, and ease of use.Airflow :

Airflow :

• - Use Apache Airflow to orchestrate and schedule complex data workflows.
• - Develop and maintain DAGs for task automation and monitoring.

Kafka :

• - Utilize Apache Kafka for building real-time data streaming pipelines.
• - Integrate Kafka with other systems for event-driven data processing.Big Data Technologies :

Big Data Technologies :

• - Experience with big data processing frameworks such as Hive, EMR, and Databricks.
• - Familiarity with data storage solutions like Snowflake for analytics and reporting.Data Lakes and Data Warehousing :

Data Lakes and Data Warehousing :

• - Work with data lake solutions and data warehouses for storing and managing large datasets.
• - Implement data lake architectures for scalable data ingestion and processing.

Technical Knowledge and Skills Required :

• - Strong hands-on experience with PySpark or Spark with Scala for big data processing.
• - Proficient in writing complex SQL queries and optimizing them for performance.
• - Understanding of ETL/ELT design principles and best practices.
• - Basic to intermediate knowledge of CI/CD processes and tools.
• - Familiarity with cloud platforms (AWS, Azure, Google Cloud) and their data -related services.
• - Ability to work independently and resolve issues effectively.

Soft Skills Required :

• - Strong analytical and problem‑solving skills.
• - Excellent communication and collaboration abilities.
• - Ability to manage multiple tasks and projects efficiently.
• - A proactive attitude towards learning new technologies and improving existing skills.

Work Experience :

• - 5+ years of relevant experience in data engineering, focusing on big data processing and cloud technologies.
• - Experience working with data engineering tools and frameworks like Airflow, Kafka, Hive, EMR, Databricks, and Snowflake.Education :

Education :

• - Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field.
• - A Master's degree is a plus.