Databricks Specialist

Tata Consultancy Services Limited; New York (NY); Presencial; USD 130.000 - 150.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Discretionary Annual Incentive

Comprehensive Medical Coverage

Parental Leaves

401K Plan

Certification & Training Reimbursement

Vacation & Holidays

Descripción de la vacante

Tata Consultancy Services Limited in New York, NY is seeking an experienced Engineer to lead data architecture and cloud data platform initiatives. You will design scalable Databricks pipelines, collaborate with cross-functional teams, and ensure security, governance and performance across analytic workloads.

The role requires 8+ years in data engineering or architecture, strong SQL, PySpark, and experience with AWS data services.

Formación

• Bachelor's degree in CS or equivalent with 8+ years of experience.
• Strong experience in data architecture and large-scale cloud data platforms.
• Proficiency in Databricks, PySpark, SQL and data governance.

Responsabilidades

• Architect scalable data processing frameworks using Databricks for analytical workloads.
• Design ETL/ELT pipelines for ingestion, transformation, and distribution of data.
• Implement data governance, security and privacy controls across platforms.
• Develop CI/CD workflows and IaC practices for data pipelines and environments.
• Collaborate with product owners and engineers to translate requirements into architecture.
• Mentor team members on Spark optimization and cloud-native design patterns.

Conocimientos

AWS

Python

Educación

Bachelor's degree in computer science or similar field

Herramientas

Databricks

Unity Catalog

AWS IAM

Descripción del empleo

Job Description

Must Have Technical/Functional Skills

• 8+ years of experience in data architecture, data engineering or database design and large-scale enterprise cloud data implementations in complex, highly regulated environments with
• Databricks platform. Deep understanding of modern data technology stacks, cloud data platforms (AWS preferred), and enterprise software solutions.
• 5+ years of experience in Tableau visualization preferred.
• Demonstrated experience designing, architecting and supporting applications with a strong emphasis on scalability, security, availability and performance. Demonstrated knowledge of and leading adoption of industry best practices in the areas of DataOps and modern data stack tools, data governance, SQL and other query tools.
• Demonstrated experience in cloud data architecture skills including designing, architecting and delivering modern cloud data platforms to ingest, process, store and expose data across the enterprise.
• Expert knowledge of data modeling techniques and tools, data lifecycle management and integration patterns to ensure that data flows are consistent, reliable and reusable across services and solutions. Experience with data storage technologies, partitioning strategies, caching, data access patterns, and robust security strategies enforcing privacy and regulatory requirements including designing standardized approaches for data classification, access control, encryption, retention and auditability. Knowledge of data lake houses, data warehousing concepts and data governance.
• Outstanding communication and collaboration skills, including translating technical topics and risks into business teams. Ability to personalize communication to audience from technologists to executive leadership.
• Extensive, in-depth experience implementing highly complex technology projects in a cross-functional matrix environment with demonstrated ability to drive consensus and deliver results.
• Examples of leading through influence and successfully resolving conflicting priorities are required.
• Bachelor's degree in computer science or similar field with 8+ years of related work experience.

Responsibilities

Architect scalable data processing frameworks leveraging Databricks platform for analytical workloads Implement cost-optimization strategies balancing compute resources, storage tiering, and cluster configurations Develop and maintain coding standards for PySpark/Python within Databricks notebooks ensuring maintainability and performance Design and architect ETL/ELT pipelines supporting ingestion, transformation, and distribution of analytic datasets. Design Databricks workspace architectures integrating Unity Catalog for centralized governance, AWS IAM for authentication, and cross-account data access patterns. Design data quality frameworks and validation logic within processing workflows Understanding of complex SQL queries and Spark transformations for data manipulation, aggregation, and analytical computations Establish patterns for incremental processing, change data capture, and historical data management Architect MLOps pipelines for model deployment and monitoring using AWS services, Domino Data Lab, and related platforms Implement CI/CD workflows using GitLab for data pipeline deployment and version control Establish infrastructure-as-code practices for environment consistency and disaster recovery Establish data governance frameworks within centralized data lake environment Design dimensional data models and semantic layers optimized for Tableau and BI consumption Explore and implemen t generative AI capabilities Partner with product owners, architects, data engineers, analysts, and application teams to translate requirements into technical architectures Mentor team members on best practices for Spark optimization, data modeling, and cloud-native design patterns Evaluate emerging technologies and platforms for potential adoption into the data ecosystem Document architectural decisions, design patterns, and technical standards

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Location

New York, NY

Function

TECHNOLOGY

Role

Engineer

ID

434332

Skills

AWS | Python

Salary Range

$130,000 - $150,000 a year

Salary Range

$130,000 - $150,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE