Databricks Engineer

Tata Consultancy Services; Raleigh (NC); On-site; USD 100,000 - 140,000; Full time

Job summary

Tata Consultancy Services in Raleigh, NC is seeking a Data Engineer to design, develop, and operate scalable ETL/ELT pipelines using Databricks, Spark, and PySpark. You will build data transformations and analytics solutions in SQL and Delta Lake, collaborating with product owners, data architects, analysts, and business teams.

You'll ensure robust data operations on Azure-based cloud platforms, implement orchestration and data quality controls, and maintain CI/CD pipelines for notebooks and

Qualifications

• Bachelor's degree in Computer Science or related field.
• 5+ years of experience in data engineering or ETL development.
• Strong SQL, Python/Scala, and analytical skills.

Responsibilities

• Design, develop, and deploy scalable ETL/ELT pipelines using Databricks, Spark, and PySpark.
• Build and optimize SQL-based data transformation and analytics solutions.
• Collaborate with product owners, data architects, analysts, and business teams on data models and workflows.
• Develop Delta Lake structures and enterprise data pipelines.
• Implement job orchestration, monitoring, and data quality controls.
• Maintain CI/CD pipelines for Databricks notebooks, jobs, and workflows.
• Support Azure-based cloud data platforms and ensure reliable, high-performing data operations.

Skills

Databricks

Apache Spark

SQL

ETL/ELT development

Data modeling

Python or Scala

Azure cloud platforms

Production support and monitoring

Education

Bachelor's degree in Computer Science

Tools

Databricks

Apache Spark

Job description

Job Description

• Production support and operational monitoring
• Design, develop, and deploy scalable ETL/ELT pipelines using Databricks, Spark, and PySpark.
• Build and optimize SQL-based data transformation and analytics solutions.
• Collaborate with product owners, data architects, analysts, and business teams on data models and workflows.
• Develop Delta Lake structures and enterprise data pipelines.
• Implement job orchestration, monitoring, and data quality controls.
• Maintain CI/CD pipelines for Databricks notebooks, jobs, and workflows.
• Support Azure-based cloud data platforms and ensure reliable, high-performing data operations.

Must Have Technical/Functional Skills

• Databricks and Apache Spark
• SQL and data analysis
• ETL/ELT development and optimization
• Data modeling and performance tuning
• Python or Scala
• Azure cloud-based data platforms
• Production support and operational monitoring

Roles & Responsibilities

• Design, develop, and deploy scalable ETL/ELT pipelines using Databricks, Spark, and PySpark.
• Build and optimize SQL-based data transformation and analytics solutions.
• Collaborate with product owners, data architects, analysts, and business teams on data models and workflows.
• Develop Delta Lake structures and enterprise data pipelines.
• Implement job orchestration, monitoring, and data quality controls.
• Maintain CI/CD pipelines for Databricks notebooks, jobs, and workflows.
• Support Azure-based cloud data platforms and ensure reliable, high-performing data operations.

Salary Range

$100,000 to $140,000 per year

Qualifications

BACHELOR OF COMPUTER SCIENCE