Python Hadoop Engineer

Tata Consultancy Services; Charlotte (NC); On-site; USD 100,000 - 110,000; Full time

Benefits offered by this job

Discretionary annual incentive

Comprehensive medical coverage

Family leaves

Commuter benefits

Certification & training reimbursement

Vacation & holidays

401K plan

Job summary

Tata Consultancy Services (TCS) in Charlotte, NC is seeking a Data Engineer with 5–10 years of experience to design and optimize ETL/ELT pipelines, work with Spark, Hadoop ecosystems, and cloud data platforms. Strong Python/SQL skills and data governance experience are essential.

You will collaborate with engineering, product, and business teams to ensure data quality, observability, and secure data management across enterprise datasets in a regulated environment.

Qualifications

• 5–10 years of experience in Data Engineering, Software Engineering, or related technical discipline.
• Strong proficiency in Python and SQL for advanced data transformations.
• Hands-on experience designing and building ETL/ELT pipelines and data ingestion processes.
• Experience with distributed data processing tools such as Apache Spark, Databricks, or Hadoop ecosystems.
• Experience with datasets in relational and cloud-based platforms (Teradata, Snowflake, SQL Server, Azure/AWS/GCP).
• Understanding of data modeling, metadata, data quality controls, data lineage, and secure data management.

Responsibilities

• 5–10 years of experience in Data Engineering or related field.
• Develop and optimize Python and SQL for data transformations.
• Design and implement ETL/ELT pipelines and data ingestion processes.
• Work with distributed data tools like Spark, Databricks, or Hadoop ecosystems.
• Build and manage datasets across relational and cloud platforms.
• Contribute to automated test suites and support CI practices.

Skills

PySpark

Hive

Python

SQL

Hadoop

Unix

Agile

Base support

Education

Bachelor of Computer Science

Tools

Git

Databricks

Job description

Job Description

Must Have Technical/Functional Skills

• Primary Skill: PySpark, Hive, Python, SQL, Hadoop
• Secondary: Unix, Agile, Base support
• Experience: 5 to 10 years

Roles & Responsibilities

• 5 to 10 years of experience in Data Engineering, Software Engineering, or related technical discipline.
• Strong proficiency in Python, and SQL for advanced data transformations.
• Hands-on experience designing and building ETL/ELT pipelines, data ingestion processes, and distributed data processing jobs.
• Practical experience working with distributed data tools such as Apache Spark, Databricks, or Hadoop ecosystems.
• Experience building and managing datasets in relational and/or cloud-based data platforms (Teradata, Snowflake, SQL Server, Azure/AWS/GCP).
• Solid understanding of data modeling, metadata, data quality controls, data lineage, and secure data management.
• Experience contributing to automated test suites, analyzing test failures, and supporting test-driven development.
• Knowledge of CI/CD pipelines, version control (Git), and automated deployment practices.
• Experience adhering to enterprise data governance, compliance, and operational risk frameworks.
• Ability to troubleshoot pipeline issues, performance bottlenecks, and data discrepancies.
• Strong communication skills and ability to collaborate across engineering, product, and business teams.
• Experience implementing monitoring and observability for data pipelines (logs, metrics, health checks).
• Advanced experience with performance tuning of SQL, Spark, or distributed data workflows.
• Knowledge of data security practices (encryption, masking, PII handling).
• Experience supporting analytical workloads, BI tools, or data science teams.
• Prior experience in a financial institution or other regulated industry.

A Data Engineer is essential to implement efficient data flows, enforce data management standards, enhance data quality, and support continuous delivery and release cycles. Without this role, the project risks delays in data readiness, gaps in data compliance, reduced quality of analytical outputs, and inability to support downstream systems effectively.

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
• Salary Range: $100,000 - 110,000 a year

Qualifications

BACHELOR OF COMPUTER SCIENCE