Senior PySpark Data Engineer - Hadoop & ETL

Tata Consultancy Services; Charlotte (NC); On-site; USD 100,000 - 110,000; Full time

Benefits offered by this job

Discretionary annual incentive

Comprehensive medical coverage

Family leaves

Commuter benefits

Certification & training reimbursement

Vacation & holidays

401K plan

Job summary

Tata Consultancy Services (TCS) in Charlotte, NC is seeking a Data Engineer with 5–10 years of experience to design and optimize ETL/ELT pipelines, work with Spark, Hadoop ecosystems, and cloud data platforms. Strong Python/SQL skills and data governance experience are essential.

You will collaborate with engineering, product, and business teams to ensure data quality, observability, and secure data management across enterprise datasets in a regulated environment.

Qualifications

• 5–10 years of experience in Data Engineering, Software Engineering, or related technical discipline.
• Strong proficiency in Python and SQL for advanced data transformations.
• Hands-on experience designing and building ETL/ELT pipelines and data ingestion processes.
• Experience with distributed data processing tools such as Apache Spark, Databricks, or Hadoop ecosystems.
• Experience with datasets in relational and cloud-based platforms (Teradata, Snowflake, SQL Server, Azure/AWS/GCP).
• Understanding of data modeling, metadata, data quality controls, data lineage, and secure data management.

Responsibilities

• 5–10 years of experience in Data Engineering or related field.
• Develop and optimize Python and SQL for data transformations.
• Design and implement ETL/ELT pipelines and data ingestion processes.
• Work with distributed data tools like Spark, Databricks, or Hadoop ecosystems.
• Build and manage datasets across relational and cloud platforms.
• Contribute to automated test suites and support CI practices.

Skills

PySpark

Hive

Python

SQL

Hadoop

Unix

Agile

Base support

Education

Bachelor of Computer Science

Tools

Git

Databricks

Job description

Tata Consultancy Services (TCS) in Charlotte, NC is seeking a Data Engineer with 5–10 years of experience to design and optimize ETL/ELT pipelines, work with Spark, Hadoop ecosystems, and cloud data platforms. Strong Python/SQL skills and data governance experience are essential.

You will collaborate with engineering, product, and business teams to ensure data quality, observability, and secure data management across enterprise datasets in a regulated environment.