Senior Data Engineer

Anblicks Inc.; Hyderabad; On-site; INR 2,300,000 - 3,000,000; Full time

Job summary

Anblicks Inc. in Hyderabad is seeking a Senior Data Engineer to design and build scalable data platforms and Lakehouse solutions using Databricks, PySpark, Python, SQL and cloud technologies.

You will collaborate with architects, data scientists, and business stakeholders to optimize data pipelines, implement governance and security controls, and advance DataOps and Data Product practices, while embracing CI/CD and IaC.

Qualifications

• Bachelors or masters in CS/Engineering/IS or related field.
• 6–8 years data engineering experience with 5+ years on Databricks.
• Strong Python, PySpark, SQL, and distributed data processing.
• Experience with cloud platforms (AWS/Azure/GCP).

Responsibilities

• Design and build scalable data pipelines using Databricks and Spark.
• Develop ETL/ELT workflows and data models for analytics, ops, and AI/ML use cases.
• Establish engineering standards, design patterns, and best practices.
• Optimize performance via Spark tuning and partitioning.
• Implement data quality, observability, governance, and security.
• Follow CI/CD, automated testing, IaC, and DevSecOps.
• Lead root-cause analysis and improve platform reliability.

Skills

Databricks

PySpark

Python

SQL

Spark optimization

Data modeling

Data governance

CI/CD

Data quality

Education

Bachelor's or Master's degree in Computer Science / Engineering / Information Systems

Tools

Terraform

CloudFormation

Delta Lake

Unity Catalog

Databricks Asset Bundles (DABs)

Job description

Anblicks is a Great Place to Work® Certified Data & AI company helping enterprises build modern, intelligent, and scalable data platforms. As a Snowflake Elite Services Partner, Databricks Consulting Partner, AWS Advanced Consulting Partner, and Microsoft Azure Gold Partner, we work with leading organizations to solve complex data, cloud, analytics, and AI challenges.

We are looking for an experienced Senior Data Engineer to design and build scalable data platforms and high-performance data solutions using Databricks, PySpark, Python, SQL, and cloud technologies.

Role Summary

As a Senior Data Engineer, you will be responsible for designing, developing, and optimizing modern data engineering solutions and Lakehouse platforms. You will work on large-scale data processing, data pipelines, data modeling, performance optimization, governance, and cloud-native technologies while contributing to technical design and engineering best practices.

Key Responsibilities

• Design and develop scalable, reliable, and high-performance data pipelines using Databricks, Spark, PySpark, and cloud technologies.
• Design and implement complex ETL/ELT workflows using Databricks, PySpark, SQL, and cloud services.
• Establish engineering standards, design patterns, and best practices for data engineering solutions.
• Make technical and architectural recommendations focused on scalability, performance, maintainability, security, and cost optimization.
• Develop reusable frameworks, libraries, and accelerators to improve engineering productivity.
• Design and implement optimized data models for analytics, operational, and AI/ML use cases.
• Identify and resolve performance bottlenecks through Spark optimization, query tuning, partitioning, and resource management.
• Implement and support data quality, monitoring, observability, governance, and security capabilities.
• Follow software engineering best practices including code reviews, automated testing, CI/CD, Infrastructure as Code, and DevSecOps.
• Lead root-cause analysis and resolution of complex production issues.
• Drive improvements in platform reliability, resiliency, scalability, and operational efficiency.
• Contribute to modern DataOps and Data Product practices.
• Collaborate with architects, developers, data scientists, business stakeholders, and cross-functional teams.

Required Skills & Experience

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related field.
• 6–8 years of overall Data Engineering experience, including 5+ years of hands-on Databricks experience.
• Strong experience building large-scale data platforms on public cloud environments.
• Expert-level proficiency in Python, PySpark, SQL, and distributed data processing.
• Strong understanding of Lakehouse architecture, data warehousing, and modern data engineering concepts.
• Hands-on experience with Spark optimization, performance tuning, partitioning strategies, and workload management.
• Strong experience with AWS services such as S3, EMR, Glue, Lambda, ECS/EKS, Redshift, and IAM.
• Experience designing scalable data models and data architectures for analytical and operational workloads.
• Strong understanding of CI/CD, automated testing, and software engineering best practices.
• Hands-on experience with Databricks Asset Bundles (DABs).
• Experience implementing security, governance, privacy, and compliance controls within data platforms.
• Ability to drive technical initiatives and contribute to architectural decisions.
• Strong problem-solving, communication, and stakeholder management skills.
• Experience working in Agile, cross-functional, and globally distributed teams.

Good to Have

• Experience with Infrastructure as Code tools such as Terraform or CloudFormation.
• Experience implementing data quality, observability, and metadata management solutions.
• Experience with streaming technologies such as Kafka, Azure Event Hubs, or Amazon Kinesis.
• Experience deploying and operationalizing Machine Learning and Generative AI solutions.
• Strong knowledge of Delta Lake optimization, Unity Catalog, and Databricks governance.
• Experience leading technical design reviews and mentoring engineering teams.

Why Anblicks?

• Opportunity to work with leading cloud and data technologies including Databricks, Snowflake, AWS, Azure, and AI/ML.
• Exposure to enterprise-scale transformation and modernization initiatives.