Data Engineer

Tata Consultancy Services; Indore District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Indore seeks an AWS Databricks Engineer to design, build, and optimize a scalable data platform on the Databricks Lakehouse architecture on AWS.

The role requires 3+ years of hands-on experience with Databricks, AWS data services, and strong Python/SQL skills. You will implement end-to-end data pipelines, ensure data reliability, and contribute to security/governance using Unity Catalog and IAM.

Qualifications

• 3+ years of hands-on experience building production-grade data pipelines on Databricks and AWS.
• Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.

Responsibilities

• Design and deploy robust, scalable data pipelines (ETL/ELT) using Databricks and AWS services.
• Implement Lakehouse architectures with Delta Lake for reliability and high performance.
• Optimize Spark jobs for performance, cost, and memory management in Databricks clusters.
• Manage and integrate Databricks with AWS services like S3, Glue, IAM, EMR, RDS, Redshift.
• Implement security and governance controls including Unity Catalog and IAM roles.

Skills

Databricks','weight

Education

Bachelor's degree in CS/IT/Data Engineering

Tools

Terraform

Git

Job description

We have Organised a Walkin drive for Below post at Indore office-

Job Title: AWS Databricks Engineer

Exp - 5+ yeas

We are seeking a skilled Data Engineer with deep expertise in AWS and Databricks to design, build, and optimize our next-generation data platform. In this role, you will implement scalable data pipelines, manage cloud infrastructure, and enable advanced analytics and machine learning capabilities. You will bridge the gap between raw data and actionable business insights by leveraging the power of the Databricks Lakehouse architecture on AWS.

• Design and deploy robust, scalable data pipelines (ETL/ELT) using Databricks, Apache Spark (Python/Scala/SQL), and AWS services.
• Implement Lakehouse architectures utilizing Delta Lake to ensure data reliability, ACID transactions, and high performance.
• Optimize Spark jobs for performance, cost, and memory management within Databricks clusters.
• Manage and integrate Databricks with AWS native services including Amazon S3, AWS Glue, IAM, Amazon EMR, RDS, and Redshift.
• Implement security and governance controls, ensuring secure data access via Unity Catalog, IAM roles, and encryption (KMS).

Operations & DevOps

• Automate deployments and workflows using CI/CD pipelines (e.g., GitHub Actions, GitLab, Jenkins) and Infrastructure as Code (Terraform).
• Orchestrate workflows using Databricks Workflows, Apache Airflow, or AWS Step Functions.
• Monitor platform health, cluster utilization, and cost optimization metrics to ensure efficient cloud spending.

Technical Skills

• Databricks Expertise: Strong experience with Databricks runtimes, Delta Lake, Unity Catalog, Photon engine, and Databricks SQL.
• AWS Cloud: Proficient with core AWS data and infrastructure services (S3, IAM, VPC, EC2, Lambda, Glue).
• Programming: Advanced programming skills in Python (PySpark) or Scala, along with expert-level SQL.
• Data Modeling: Solid understanding of data warehousing concepts, star/snowflake schemas, and dimensional modeling.
• DevOps/IaC: Experience with Terraform and version control (Git) for managing data infrastructure.

Experience & Qualifications

• Education: Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Experience: 3+ years of hands-on experience building production-grade data pipelines on Databricks and AWS.
• Certifications (Preferred):
• Databricks Certified Data Engineer Professional / Associate
• AWS Certified Data Engineer - Associate or AWS Solutions Architect