Sr. Data Engineer - Databricks

Anblicks Inc.; Hyderabad; On-site; INR 3,000,000 - 5,500,000; Full time

Job summary

Anblicks Inc. in Hyderabad is seeking a Senior Data Engineer to design, build, and optimize scalable data platforms and high-performance ETL/ELT pipelines using Databricks, Apache Spark, Python, Java, and SQL.

The role requires strong hands-on experience in enterprise-grade data solutions, processing large datasets, and implementing reliable data ingestion frameworks. You will collaborate with data, cloud, analytics, and engineering teams to deliver scalable solutions.

Qualifications

• 6-10 years of professional experience in software development and/or data engineering, with proven experience delivering production-grade data solutions.
• Strong hands-on experience in Data Engineering and ETL/ELT development.
• Strong experience with Databricks and Apache Spark.
• Strong programming skills in Python and Java.
• Advanced proficiency in SQL, including query optimization and working with large datasets.
• Experience designing and developing scalable data pipelines and distributed data processing solutions.
• Strong understanding of data warehousing, data modeling, and large-scale data processing.
• Hands-on experience with data ingestion, transformation, data quality, and pipeline optimization.
• Strong troubleshooting, debugging, and performance tuning capabilities.
• Experience working in Agile development environments and collaborating with cross-functional teams.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark.
• Build robust data ingestion, transformation, processing, and validation frameworks for large-scale datasets.
• Develop and optimize data processing workloads using Python, Java, Spark, and SQL.
• Work extensively with Databricks to develop scalable data engineering solutions and production-grade pipelines.
• Implement efficient data transformation and storage solutions using modern data engineering practices.
• Write complex and optimized SQL queries for large-volume data processing and analytics.
• Perform performance tuning and optimization of Spark jobs, SQL queries, pipelines, and data processing workloads.
• Implement data quality checks, validation frameworks, monitoring, logging, and error-handling mechanisms.
• Work with stakeholders, data analysts, architects, and engineering teams to understand requirements and deliver effective data solutions.
• Contribute to the design and implementation of cloud-based data platforms and modern data architectures.
• Troubleshoot production issues and ensure the reliability, scalability, and performance of data pipelines.
• Follow data engineering best practices related to security, governance, data quality, and operational excellence.
• Mentor junior engineers and contribute to technical standards, reusable frameworks, and engineering best practices.

Skills

ETL/ELT development

Python

Java

SQL

Data engineering

Data ingestion

Data quality

Performance tuning

Agile

Distributed processing

Education

Bachelor's degree in Computer Science/Computer Engineering/Software Engineering

Tools

Databricks

Apache Spark

Delta Lake

Databricks Workflows

CI/CD

Git

AWS

Job description

Experience: 6+ Years

Number of Positions: 6

Role Overview

Anblicks is looking for experienced Senior Data Engineers to design, build, and optimize scalable data platforms and high-performance ETL/ELT pipelines using Databricks, Apache Spark, Python, Java, and SQL.

The ideal candidate will have strong hands-on experience in building enterprise-grade data solutions, processing large-scale datasets, optimizing distributed data workloads, and implementing reliable data ingestion and transformation frameworks. The role will involve working closely with data, cloud, analytics, and engineering teams to deliver scalable and high-quality data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark.


• Build robust data ingestion, transformation, processing, and validation frameworks for large-scale datasets.


• Develop and optimize data processing workloads using Python, Java, Spark, and SQL.


• Work extensively with Databricks to develop scalable data engineering solutions and production-grade pipelines.


• Implement efficient data transformation and storage solutions using modern data engineering practices.


• Write complex and optimized SQL queries for large-volume data processing and analytics.


• Perform performance tuning and optimization of Spark jobs, SQL queries, pipelines, and data processing workloads.


• Implement data quality checks, validation frameworks, monitoring, logging, and error-handling mechanisms.


• Work with stakeholders, data analysts, architects, and engineering teams to understand requirements and deliver effective data solutions.


• Contribute to the design and implementation of cloud-based data platforms and modern data architectures.


• Troubleshoot production issues and ensure the reliability, scalability, and performance of data pipelines.


• Follow data engineering best practices related to security, governance, data quality, and operational excellence.


• Mentor junior engineers and contribute to technical standards, reusable frameworks, and engineering best practices.

Required Qualifications

• 6 - 10 years of professional experience in software development and/or data engineering, with proven experience delivering production-grade data solutions.


• Strong hands-on experience in Data Engineering and ETL/ELT development.


• Strong experience with Databricks and Apache Spark.


• Strong programming skills in Python and Java.


• Advanced proficiency in SQL, including query optimization and working with large datasets.


• Experience designing and developing scalable data pipelines and distributed data processing solutions.


• Strong understanding of data warehousing, data modeling, and large-scale data processing.


• Hands-on experience with data ingestion, transformation, data quality, and pipeline optimization.


• Strong troubleshooting, debugging, and performance tuning capabilities.


• Experience working in Agile development environments and collaborating with cross-functional teams.

Preferred Qualifications

• Experience working with AWS cloud services and cloud-based data platforms.


• Experience with Delta Lake and modern Databricks data engineering capabilities.


• Knowledge of Databricks Workflows, Jobs, notebooks, clusters, and deployment practices.


• Experience with CI/CD, Git, and DevOps practices for data engineering.


• Exposure to AdTech, MarTech, Customer Data Platforms (CDP), or customer analytics environments.


• Experience working with high-volume, complex, and diverse datasets.

Education

Bachelor’s degree in Computer Science, Computer Engineering, Software Engineering, or a related technical field, or equivalent professional experience.