Databrick developer

Tata Consultancy Services; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced Databricks Developer/Lead to design, develop, and optimize data solutions using Databricks, Spark, and Delta Lake. You will build data pipelines, lakes, and warehouses while collaborating with data scientists on ML deployments.

The role emphasizes performance optimization of Spark jobs and notebooks and guiding cross-functional teams to ensure data quality and scalable data architectures.

Qualifications

• Experience with Databricks, Spark, Delta Lake and related technologies.
• Strong programming skills in Python, Scala or Java.
• Experience in data warehousing, ETL and data pipelines.
• Proficiency in SQL and Unity Catalogue, Workflows/Jobs.
• Experience in data science and ML workflows is a plus.

Responsibilities

• Design, develop and optimize data solutions using Databricks and Spark.
• Develop and maintain data pipelines, data lakes, and data warehouses with Delta Lake.
• Collaborate with data scientists to deploy ML models and optimize workflows.
• Optimize Spark jobs and notebooks for performance and scalability.
• Collaborate with cross-functional teams to ensure data quality and governance.

Skills

Data engineering

Data science

SQL

ETL

Tools

Databricks

Spark

Delta Lake

Python

Scala

Java

Unity Catalogue

Workflows/Jobs

Job description

An experienced Databricks Developer / Lead to design, develop, and optimize data solutions using Databricks, Spark, and related technologies. The ideal candidate should have a strong background in data engineering, data science, and programming languages like Python, Scala, or Java.

Responsibilities:

• Data Solution Development: Design, develop, and optimize data solutions using Databricks, Spark, and related technologies.
• Data Engineering: Develop and maintain data pipelines, data lakes, and data warehouses using Databricks, Delta Lake, and other tools.
• Data Science: Collaborate with data scientists to deploy machine learning models and optimize data science workflows.
• Performance Optimization: Optimize Spark jobs, Databricks notebooks, and data pipelines for performance and scalability.
• Collaboration: Work with cross-functional teams to identify and prioritize project requirements, provide technical guidance, and ensure data quality

Required Skills:

• Databricks: In-depth knowledge of Databricks, Spark, Delta Lake, and related technologies.
• Programming: Strong programming skills in Python, Scala, or Java.
• Data Engineering: Experience with data engineering, data warehousing, and ETL tools.
• Expertise in SQL, Unity Catalogue, Workflows/Jobs
• Data Science: Experience with data science, machine learning, and statistical modeling.