Production Data Engineer: Spark, ETL & Data Ops

Tata Consultancy Services; Charlotte (NC); On-site; USD 95,000 - 115,000; Full time

Job summary

Tata Consultancy Services in Charlotte, NC is seeking a Data Engineer to support production data pipelines and data platforms. You will work across Hadoop, Spark, and cloud-based data stores to ensure data readiness and reliable analytics.

The role emphasizes ETL/ELT development, performance tuning, and monitoring, with collaboration across engineering, product, and business teams to resolve issues and improve data quality.

Qualifications

• 5 to 10 years in Data Engineering or related field.
• Strong proficiency in Python and SQL for data transformations.
• Experience designing ETL/ELT pipelines and data ingestion processes.
• Hands-on experience with distributed data tools such as Apache Spark, Databricks, or Hadoop ecosystems.

Responsibilities

• 5 to 10 years of experience in Data Engineering, Software Engineering, or related technical discipline.
• Strong proficiency in Python, and SQL for advanced data transformations.
• Hands-on experience designing and building ETL/ELT pipelines, data ingestion processes, and distributed data processing jobs.
• Practical experience working with distributed data tools such as Apache Spark, Databricks, or Hadoop ecosystems.
• Experience building and managing datasets in relational and/or cloud based data platforms (Teradata, Snowflake, SQL Server, Azure/AWS/GCP).
• Solid understanding of data modeling, metadata, data quality controls, data lineage, and secure data management.
• Experience contributing to automated test suites, analyzing test failures, and supporting test-driven development.
• Knowledge of CI/CD pipelines, version control (Git), and automated deployment practices.
• Experience adhering to enterprise data governance, compliance, and operational risk frameworks.
• Ability to troubleshoot pipeline issues, performance bottlenecks, and data discrepancies.
• Strong communication skills and ability to collaborate across engineering, product, and business teams.
• Experience implementing monitoring and observability for data pipelines (logs, metrics, health checks).
• Advanced experience with performance tuning of SQL, Spark, or distributed data workflows.
• Knowledge of data security practices (encryption, masking, PII handling).
• Experience supporting analytical workloads, BI tools, or data science teams.
• Prior experience in a financial institution or other regulated industry

Skills

PySpark

Hive

Python

SQL

Hadoop

Unix

Agile

Base support

Job description

Tata Consultancy Services in Charlotte, NC is seeking a Data Engineer to support production data pipelines and data platforms. You will work across Hadoop, Spark, and cloud-based data stores to ensure data readiness and reliable analytics.

The role emphasizes ETL/ELT development, performance tuning, and monitoring, with collaboration across engineering, product, and business teams to resolve issues and improve data quality.