Data Bricks Data Engineer

Tata Consultancy Services; Seattle (WA); On-site; USD 120,000 - 140,000; Full time

Benefits offered by this job

Annual Discretionary Incentive

Comprehensive Medical Coverage

401K Plan

Vacation and Sick Leave

Certification and Training Reimbursement

Job summary

Tata Consultancy Services is looking for a Data Engineer to design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python, ensuring collaboration with stakeholders to gather business requirements and implementing data governance standards. The ideal candidate should possess strong skills in Python, PySpark, and Azure Data Bricks, along with experience in data integration tools such as IBM Data Stage.

The base salary is competitive, ranging from $120,000 to $140,000 annually, with excellent benefits including medical coverage, time off, and legal assistance support.

Qualifications

• Experience with data integration tools like IBM Data Stage.
• Strong programming skills in Python and PySpark.
• Proven experience building and optimizing complex data pipelines in Azure.

Responsibilities

• Design, develop, and maintain ETL/ELT workflows using PySpark and Python.
• Implement and monitor data processing workloads in Azure Databricks.
• Collaborate with stakeholders to gather business requirements.
• Author Infrastructure-as-Code for Azure resources and deploy consistent environments.

Skills

IBM Data Stage

Python

PySpark

SQL

Azure Data Bricks

Terraform

GitLab

Azure DevOps

Agile methodologies

Containerization

Education

Bachelor of Computer Science

Tools

Azure Services

Databricks

ARM templates

Job description

Job Description

• Awareness of IBM Data Stage ETL/ELT data integration tool to understand existing code.
• Develop, Test, Deploy, Optimize, and monitor large-scale data processing workloads in Azure Data Bricks ETL.
• Ensure and lead the efforts to review Legacy Data Stage legacy code and migrated Data bricks code to ensure functionality is not deviated.
• Strong programming skills in Python and PySpark.
• Advanced proficiency writing SQL for analytics and ETL processes.
• Proven experience building and optimizing complex data pipelines in Azure.
• Hands-on experience with Azure Databricks: cluster management, job scheduling, workspace governance.
• Strong working knowledge of core Azure services: Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, service principals, and managed identities.
• Experience with container services (ACA, container instances) and containerized data workloads.
• Familiarity with Azure networking concepts and secure network integration for data platforms.
• Experience creating Azure infrastructure using ARM templates.
• Proficient with GitLab and Azure DevOps for CI/CD and source control workflows.
• Strong analytical, problem-solving, and communication skills; proven ability to work cross-functionally.
• Experience working in Agile teams and understanding of data governance frameworks.
• Hands-on experience provisioning Databricks resources with Terraform; ability to author and maintain Terraform templates and modules.
• Demonstrated experience implementing cluster autoscaling and autoscaling policies through Terraform.
• Experience creating reusable Terraform modules and implementing infrastructure-as-code best practices (module structure, state management, remote backends).
• Proven experience working on Databricks platform operations, including cluster configuration, job orchestration, and platform optimization.
• Experience configuring high-availability Databricks deployments and operating across multiple availability zones/regions.
• Familiarity with Metastore/Unity Catalog configuration and metadata governance in Databricks.
• Hands-on experience building data pipelines and ingestion workflows into medallion-layer architectures (bronze/silver/gold).
• Strong scripting skills (Python, Bash, or similar) and familiarity with CI/CD for Terraform and Databricks deployments.
• Strong troubleshooting, performance tuning, and cost optimization skills.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python.
• Ensure and lead the efforts to review Legacy Data Stage legacy code and migrated Data bricks code to ensure functionality is not deviated.
• Implement, optimize, and monitor large-scale data processing workloads in Azure Databricks, including cluster configuration, autoscaling, and governance.
• Build and maintain data integration and orchestration solutions using Azure services to meet performance, availability, and security requirements.
• Collaborate with data consumers, thread authors/owners, and stakeholders to gather business requirements, prioritize needs, and translate analytical objectives into technical designs.
• Implement secure data access patterns using Azure Active Directory, Managed Identities, and service principals.
• Author Infrastructure-as-Code for Azure resources (ARM templates) and deploy consistent, repeatable environments.
• Configure and operate Azure components including Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, Azure Container Apps / container instances, and related services.
• Collaborate with networking and security teams to design and implement Azure networking for data solutions.
• Implement monitoring, alerting, and cost optimization for data workloads (Log Analytics, metrics, and dashboards).
• Use GitLab and Azure DevOps for source control, CI/CD pipelines, and release management.
• Follow Agile/Scrum practices and participate in sprint planning, standups, and retrospectives.
• Ensure solutions meet data governance, lineage, and compliance requirements.
• Operations Support and Oncall Support for Production Issues and Deployments.

Base Salary Range

$120,000 to $140,000 Per Annum

TCS Employee Benefits Summary

Discretionary Annual Incentive.

Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.

Family Support: Maternal & Parental Leaves.

Insurance Options: Auto & Home Insurance, Identity Theft Protection.

Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.

Time Off: Vacation, Time Off, Sick Leave & Holidays.

Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Qualifications

BACHELOR OF COMPUTER SCIENCE