Data & Cloud Tech Lead - Databricks & AWS

Tata Consultancy Services; Bloomfield (CA); On-site; USD 110,000 - 125,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Medical Coverage

Job summary

Tata Consultancy Services in Bloomfield, CA, is seeking a data engineer to design, build, and optimize large-scale data pipelines using Databricks and Spark. You will develop ETL/ELT processes, ingest data from multiple sources, and leverage AWS services to support cloud analytics workloads.

The role requires strong SQL, Databricks expertise, and experience with data governance, security, and Agile practices.

Qualifications

• Experience designing and implementing large-scale data pipelines.
• Strong SQL skills with complex queries and indexing.
• Proficiency with Databricks and Spark.
• Experience with AWS data services.
• Knowledge of data governance and security practices.

Responsibilities

• Design, develop, and optimize large-scale data processing solutions using Databricks.
• Build and maintain robust ETL/ELT pipelines using Python and Databricks.
• Develop scalable data ingestion, transformation, and integration frameworks from multiple data sources.
• Leverage Databricks features including Delta Lake, Spark SQL, Unity Catalog, Workflows, and ML capabilities.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Work with AWS services to support cloud-based data platforms and analytics workloads.
• Implement data quality checks, monitoring, performance tuning, and operational support processes.
• Participate in architecture discussions and contribute to best practices for data engineering and cloud adoption.
• Ensure compliance with security, governance, and data management standards.
• Actively participate in Agile ceremonies including sprint planning, backlog grooming, daily standups, sprint reviews, and retrospectives.

Skills

Databricks

Python Spark

AWS

Production experience

Query optimization

SQL

Indexing

Schema design

Education

Bachelor of Computer Science

Tools

EC2

S3

Lambda

Glue

EKS

Jenkins

GitLab

GitHub Actions

Job description

Tata Consultancy Services in Bloomfield, CA, is seeking a data engineer to design, build, and optimize large-scale data pipelines using Databricks and Spark. You will develop ETL/ELT processes, ingest data from multiple sources, and leverage AWS services to support cloud analytics workloads.

The role requires strong SQL, Databricks expertise, and experience with data governance, security, and Agile practices.