Data Engineer - Synapse Analytics

Forward Eye Technologies; Dadri; On-site; INR 1,400,000 - 2,400,000; Full time

Job summary

Forward Eye Technologies is hiring a Data Engineer to design and manage modern data pipelines in Azure. You will build and optimize ETL processes, leverage Databricks for processing, and implement robust data validation and governance practices.

The role requires collaboration with data scientists and business stakeholders to deliver reliable analytics solutions. Ideal candidates have 5+ years in data engineering, strong Python scripting skills, and hands-on experience with Azure Data Factory,

Qualifications

• Minimum of 5 years of experience in data engineering.
• Strong hands-on experience with Azure Data Factory for data integration and orchestration.
• Proven expertise in using Databricks for data processing and analytics.
• Proficiency in Python scripting for data processing and automation.

Responsibilities

• Design, develop, and maintain ETL pipelines using Azure Data Factory to integrate data from multiple sources into the data lake or warehouse.
• Develop and manage scalable data processing pipelines using Databricks.
• Write and optimize PySpark scripts for data transformation, aggregation, and analysis.
• Design data models and storage solutions in Azure Data Lake, Azure SQL Data Warehouse, and related services.
• Collaborate with data scientists and analysts to translate data requirements into technical solutions.
• Monitor, troubleshoot, and optimize data pipelines for performance, cost, and reliability.
• Implement data validation, error handling, and security best practices within the Azure environment.

Skills

Data engineering

Problem-solving

Communication

Independent work

Tools

Azure Data Factory

Databricks

PySpark

Python

Git & CI/CD

Job description

Job Responsibilities :

Data Integration and Orchestration :

• Design, develop, and maintain ETL pipelines using Azure Data Factory (ADF) to integrate data from various sources into the data lake or data warehouse.
• Implement data workflows and orchestrate data movement and transformation processes within the ADF environment.

Data Processing and Analytics :

• Utilize Databricks to develop and manage scalable data processing pipelines.
• Write and optimize PySpark scripts for data transformation, aggregation, and analysis within Databricks.
• Perform real-time and batch data processing on large datasets to support analytics and reporting needs.

Python Scripting and Automation :

• Develop Python scripts to automate data processing tasks, enhance data pipelines, and integrate with other Azure services.
• Implement data validation and error-handling mechanisms within Python scripts to ensure data accuracy and reliability.

Data Modeling and Storage :

• Design and implement data models that support efficient querying, storage, and retrieval of data in the Azure ecosystem.
• Optimize data storage solutions within Azure, including Azure Data Lake, Azure SQL Data Warehouse, and other related services.

Collaboration and Stakeholder Engagement :

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Provide technical expertise and guidance on best practices for data engineering and cloud-based solutions.

Performance Tuning and Optimization :

• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, and cost efficiency.
• Implement best practices for data security, governance, and compliance within the Azure environment.

Required Qualifications :

Experience :

• Minimum of 5 years of experience in data engineering or related roles.
• Strong hands-on experience with Azure Data Factory for data integration and orchestration.
• Proven expertise in using Databricks for data processing and analytics.
• Proficiency in Python scripting, particularly in the context of data processing and automation.

Technical Skills :

• In-depth knowledge of PySpark for large-scale data processing.
• Familiarity with Azure services such as Azure Data Lake, Azure SQL Data Warehouse, and Azure Synapse Analytics.
• Experience with data modeling, ETL processes, and data architecture design in cloud environments.
• Strong understanding of CI/CD pipelines and version control using Git.

Soft Skills :

• Excellent problem-solving skills with the ability to debug and resolve complex data engineering challenges.
• Strong communication skills for effective collaboration with technical and non-technical stakeholders.
• Ability to work independently and manage multiple priorities in a fast-paced environment.

Preferred Qualifications :

• Experience with data orchestration tools such as Apache Airflow.
• Familiarity with other cloud platforms (e.g., AWS, GCP) is a plus.
• Understanding of data governance and security practices in the cloud.

Keywords :

• Azure Data Factory (ADF)
• Databricks
• PySpark
• Python
• Data Engineering
• Cloud Data Solutions
• ETL Pipelines
• Data Integration
• Data Modeling
• Azure Data Lake.

Location - Hyderabad,Ahmedabad,pune,chennai,kolkata.