Senior PySpark Platform Architect

Tata Consultancy Services; Charlotte (NC); On-site; USD 110,000 - 120,000; Full time

Benefits offered by this job

Discretionary Annual Incentive.

Comprehensive Medical Coverage

Family Support: Parental Leaves

Insurance Options: Auto & Home

Certification & Training Reimbursement

Vacation, Sick Leave & Holidays

Job summary

Tata Consultancy Services in Charlotte seeks a senior data/platform engineer with 10+ years in data engineering and platform architecture. You will design surveillance data processing platforms using Hadoop, Spark and PySpark, govern data flows, and guide cross-functional teams.

Role emphasizes building scalable ETL/ELT pipelines, real-time processing, and strong communication to influence across organizations. Regulatory-focused data platforms experience is valued.

Qualifications

• Deep Expertise in PySpark, including performance tuning and optimization.
• Strong Python development experience in large-scale distributed environment.
• Solid knowledge of Hadoop ecosystem (HDFS, Hive/Impala, YARN).
• Experience designing and governing enterprise data platforms with regulatory considerations.
• Expertise in ELT/ETL, batch and real-time data processing.
• Proficiency in Java, Scala and SQL.
• Understanding of non-functional requirements and production support.

Responsibilities

• Enterprise & Surveillance Architecture.
• Define target state architecture and roadmap for surveillance data platforms leveraging Hadoop, Spark, PySpark and Python.
• Establish standard architectural patterns for alert generation, enrichment, aggregation and reporting pipelines.
• Ensure architecture aligns with enterprise standards and regulatory expectations.
• Solution design and delivery support.
• Translate requirements into scalable technical designs for surveillance platforms.
• Review and approve detailed designs to ensure alignment with intent and standards.
• Provide hands-on architectural guidance during development, testing and implementation.
• Design and standardize Spark/PySpark frameworks for surveillance workflows.
• Modernize and optimize large-scale Hadoop workloads for performance and control.
• Implement audit-friendly lineage and governance patterns.
• Support end-to-end delivery across the software lifecycle.

Skills

PySpark

Python

Hadoop

Java/Scala/SQL

Data architecture

Communication

Education

Bachelor's degree in Computer Science

Tools

Kafka

Airflow

Oozie

Control-M

Job description

Tata Consultancy Services in Charlotte seeks a senior data/platform engineer with 10+ years in data engineering and platform architecture. You will design surveillance data processing platforms using Hadoop, Spark and PySpark, govern data flows, and guide cross-functional teams.

Role emphasizes building scalable ETL/ELT pipelines, real-time processing, and strong communication to influence across organizations. Regulatory-focused data platforms experience is valued.