Senior Databricks Engineer

Tata Consultancy Services; Malvern (Chester County); On-site; USD 120,000 - 135,000; Full time

Benefits offered by this job

Discretionary incentive

Medical coverage

Parental leaves

Insurance options

Commuter benefits

Vacation & holidays

Legal & financial assistance

Salary range

Job summary

Tata Consultancy Services in Malvern, PA is seeking a skilled data engineer to design and optimize end-to-end data pipelines using Spark, Delta Lake, and Databricks on AWS. You will build Bronze-Silver-Gold medallion lakehouse architectures, implement security with Unity Catalog and IAM, and integrate with AWS services like S3 and Redshift.

Collaborate with data scientists and analysts, drive CI/CD with Git, and apply MLflow for experiment tracking and GenAI initiatives.

Qualifications

• Bachelor's degree in Computer Science or equivalent.
• Strong Python (PySpark) and SQL skills.
• Hands-on experience with Spark, Delta Lake, and Databricks on AWS.
• Familiarity with IAM, CMK via AWS KMS, and S3 data lake.
• Experience building batch and streaming ETL pipelines.

Responsibilities

• Data Pipeline Development: ETL from multiple sources, batch and streaming.
• Spark & Code Optimization: write efficient PySpark/Scala/SQL code and optimize jobs.
• Lakehouse Management: Bronze/Silver/Gold in Delta Lake.
• Orchestration & Automation: schedule workflows with Databricks Jobs/Workflows/ DL T.
• Governance & Security: Unity Catalog, data privacy and access controls.
• Collaboration: work with Data Scientists, Analysts, and ML Engineers on features and BI/ML initiatives.

Skills

ETL/ELT pipelines

Data governance

MLOps

GenAI familiarity

Collaboration with data teams

Education

Bachelor's degree in Computer Science

Tools

Spark SQL

Delta Lake

Delta Live Tables

Databricks

Python (PySpark)

SQL

AWS S3

Amazon Redshift

IAM (AWS KMS)

Git + Databricks Git Folders

Job description

Job Description

Must Have Technical/Functional Skills

Apache Spark: Mastery of Spark SQL and the DataFrame API for distributed data processing, performance tuning, and optimizing execution plans.

Delta Lake & Lakehouse: Building medallion architectures (Bronze, Silver, Gold layers) and using optimizations like Z-Ordering and ACID transactions

Pipelines & Orchestration: Developing with Delta Live Tables (DLT) and Databricks Workflows to design resilient, idempotent ELT/ETL pipelines.

Unity Catalog: Configuring data governance, row/column-level security, and data lineage

Languages: Advanced proficiency in Python (PySpark) and SQL

Storage & Data Lakes: Expert in reading/writing to Amazon S3 and integrating with AWS data warehouses (e.g., Amazon Redshift).

Security & IAM: Implementing cross-account IAM roles, S3 bucket policies, and Customer-Managed Keys (CMK) via AWS KMS

• Compute Management: Managing cluster policies, instance profiles, and utilizing Spot Instances to optimize Databricks compute costs on AWS.

Ecosystem Integration: Familiarity with integrating Databricks jobs alongside native services like AWS Glue, Amazon Kinesis, and AWS Step Functions

CI/CD & Version Control: Git-based workflows using Databricks Git Folders and managing deployments.

AI & MLOps: Using MLflow for experiment tracking and model registries.

Familiarity with LLMs, Vector Search, and GenAI integrations.

Roles & Responsibilities

Data Pipeline Development: Extract, transform, and load (ETL) data from multiple sources, building batch and streaming pipelines.

Spark & Code Optimization: Write highly efficient PySpark, Scala, or SQL code. Troubleshoot distributed processing bottlenecks and optimize job performance.

Lakehouse Management: Work with the Medallion architecture to transition data through Bronze (raw), Silver (cleansed), and Gold (aggregated) layers in Delta Lake.

Orchestration & Automation: Schedule and automate workflows using Databricks Jobs, Workflows, or Delta Live Tables (DLT). Configure automated tests, retries, and error handling.

Governance & Security: Implement data privacy and access controls utilizing the Databricks Unity Catalog, ensuring compliance and secure data sharing.

Collaboration: Work alongside Data Scientists, Data Analysts, and ML Engineers to prepare data features and support BI reporting, ML training, and generative AI initiatives.

TCS Employee Benefits Summary

Discretionary Annual Incentive.

Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.

Family Support: Maternal & Parental Leaves.

Insurance Options: Auto & Home Insurance, Identity Theft Protection.

Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.

Time Off: Vacation, Time Off, Sick Leave & Holidays.

Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range: $120,000 - 135,000 a year

Qualifications

BACHELOR OF COMPUTER SCIENCE