Data Engineer

HGS; Hyderabad; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

HGS is seeking an experienced Data Engineer in Hyderabad to design, build and optimize scalable data pipelines and platforms using PySpark, Databricks, SQL and cloud technologies.

You will develop ETL/ELT processes, manage data lakehouse solutions, ensure data quality and governance, and collaborate with data scientists and business teams in Agile delivery.

Qualifications

• 5-8 years of experience in Data Engineering.
• Strong hands-on experience with PySpark and Databricks.
• Proficiency in Python and Advanced SQL.
• Experience with Spark, Delta Lake, and Lakehouse architecture.
• Hands-on experience with Azure Databricks, ADF, ADLS, or equivalent cloud services.
• Strong understanding of ETL/ELT frameworks and data warehousing concepts.
• Experience with data modeling and large-scale data processing.
• Knowledge of Kafka, Spark Streaming, or real-time data processing tools.
• Familiarity with Git, CI/CD pipelines, and Agile methodologies.
• Bachelor's degree in Computer Science, Engineering, or a related field; Databricks/Azure certifications preferred.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark and Databricks.
• Build and maintain data ingestion frameworks from multiple data sources.
• Develop and optimize Spark jobs for large-scale data processing.
• Design and manage data lake and lakehouse solutions.
• Implement Delta Lake and medallion architecture best practices.
• Ensure data quality, governance, and security compliance.
• Monitor, troubleshoot, and optimize data pipeline performance.
• Collaborate with Data Scientists, Analysts, and Business teams.
• Support real-time and batch data processing requirements.
• Participate in code reviews, technical design, and Agile delivery.

Skills

PySpark & Databricks

Python

Advanced SQL

Spark Streaming

Kafka

ETL/ELT concepts

Data warehousing

Agile

Git & CI/CD

Education

Bachelor's degree in Computer Science / Engineering

Databricks / Azure certifications

Tools

Azure Databricks

ADF

ADLS

Git

CI/CD

Job description

Seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines and data platforms using PySpark, Databricks, SQL, and Cloud technologies. The ideal candidate will have expertise in big data processing, ETL development, and modern data lakehouse architectures.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark and Databricks.
• Build and maintain data ingestion frameworks from multiple data sources.
• Develop and optimize Spark jobs for large-scale data processing.
• Design and manage data lake and lakehouse solutions.
• Implement Delta Lake and medallion architecture best practices.
• Ensure data quality, governance, and security compliance.
• Monitor, troubleshoot, and optimize data pipeline performance.
• Collaborate with Data Scientists, Analysts, and Business teams.
• Support real-time and batch data processing requirements.
• Participate in code reviews, technical design, and Agile delivery.

Job Requirements

• 5-8 years of experience in Data Engineering.
• Strong hands‑on experience with PySpark and Databricks.
• Proficiency in Python and Advanced SQL.
• Experience with Spark, Delta Lake, and Lakehouse architecture.
• Hands‑on experience with Azure Databricks, ADF, ADLS, or equivalent cloud services.
• Strong understanding of ETL/ELT frameworks and data warehousing concepts.
• Experience with data modeling and large-scale data processing.
• Knowledge of Kafka, Spark Streaming, or real‑time data processing tools.
• Familiarity with Git, CI/CD pipelines, and Agile methodologies.
• Bachelor's degree in Computer Science, Engineering, or a related field; Databricks/Azure certifications preferred.