Data Engineer

Tata Consultancy Services; Chennai District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Chennai seeks a Databricks Developer to design, develop, and optimize data solutions using Databricks, Spark, and Delta Lake.

You will build and maintain data pipelines and data warehouses, collaborate with data scientists to deploy ML models, and optimize Spark jobs for performance while ensuring data quality across cross-functional teams.

Qualifications

• Databricks, Spark, Delta Lake and related technologies expertise.
• Strong programming skills in Python, Scala, or Java.
• Experience with data engineering, data warehousing and ETL tools.
• Proficient in SQL, Unity Catalogue and Workflows/Jobs.
• Experience with data science, ML and statistical modeling.

Responsibilities

• Data Solution Development: Design, develop, and optimize data solutions using Databricks, Spark, and related technologies.
• Data Engineering: Develop and maintain data pipelines, data lakes, and data warehouses using Databricks, Delta Lake, and other tools.
• Data Science: Collaborate with data scientists to deploy machine learning models and optimize data science workflows.
• Performance Optimization: Optimize Spark jobs, Databricks notebooks, and data pipelines for performance and scalability.
• Collaboration: Work with cross-functional teams to identify and prioritize project requirements, provide technical guidance, and ensure data quality

Skills

Python

Scala

Java

Data engineering

Data warehousing

ETL

SQL

Data science

Machine learning

Statistical modeling

Communication

Tools

Databricks

Spark

Delta Lake

Unity Catalogue

Workflows/Jobs

Job description

Role & responsibilities

Databricks Developer

Preferred candidate profile

Responsibilities:

• Data Solution Development: Design, develop, and optimize data solutions using Databricks, Spark, and related technologies.
• Data Engineering: Develop and maintain data pipelines, data lakes, and data warehouses using Databricks, Delta Lake, and other tools.
• Data Science: Collaborate with data scientists to deploy machine learning models and optimize data science workflows.
• Performance Optimization: Optimize Spark jobs, Databricks notebooks, and data pipelines for performance and scalability.
• Collaboration: Work with cross-functional teams to identify and prioritize project requirements, provide technical guidance, and ensure data quality

Required Skills

• Databricks: In-depth knowledge of Databricks, Spark, Delta Lake, and related technologies.
• Programming: Strong programming skills in Python, Scala, or Java.
• Data Engineering: Experience with data engineering, data warehousing, and ETL tools.
• Expertise in SQL, Unity Catalogue, Workflows/Jobs
• Data Science: Experience with data science, machine learning, and statistical modeling.

Communication: Excellent communication and collaboration skills