Pyspark Developer

Tata Consultancy Services Limited

Irving (TX)

On-site

USD 100,000 - 125,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Tata Consultancy Services Limited in Irving, TX is seeking an Engineer with 7-10 years of data/ETL testing experience, hands-on PySpark, and exposure to big data tools.

You will design scalable PySpark-based test architectures, architect Hadoop-based data validation, and lead environment design, plus CI/CD pipelines and data quality systems.

Qualifications

  • 7-10 years' experience in data/ETL testing.
  • Hands-on PySpark experience.
  • Exposure to big data tools.

Responsibilities

  • Design scalable PySpark-based test architectures for ETL/data pipelines.
  • Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolution.
  • Lead system design for Hadoop/Hive test environments, including YARN resource management.

Skills

Python

Education

Bachelor of Computer Science

Job description

Job Description
Must Have Technical/Functional Skills

Associates with 7-10 years' experience in data/ETL testing with hands-on PySpark and exposure to big data tools.

Ability to perform below responsibilities:
  • 1. Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing.
  • 2. Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolution
  • 3. Lead system design for Hadoop/Hive test environments, including YARN resource management, dynamic partitioning.
  • 4. Exposure to Zephyr-Jira-ServiceNow integrated test management systems with experience on API-driven automation.
  • 5. Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments.
  • 6. Create data quality system designs using PySpark integrated with Hive metadata services.
  • 7. Design testing platforms, test data generators
  • 8. Mentor juniors on PySpark testing basics, contribute to testing strategy discussions

Salary Range: $100,000 to $125,000 per year

Location

Irving, TX

Function

TECHNOLOGY

Role

Engineer

ID

436804

Skills

Python

Salary Range

$100,000 - $125,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer – PySpark
Data Engineer – PySpark

Highbrow Technology Inc • Raleigh (NC)

Hybrid
USD 120,000 - 150,000
PySpark Data Engineer
PySpark Data Engineer

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 90,000 - 120,000
Discretionary Annual Incentive
Medical Coverage (Health, Dental & Vis
401K Plan
+1
Pyspark developer (Data engineer)
Pyspark developer (Data engineer)

MTK Technologies • Irving (TX)

On-site
USD 90,000 - 120,000
Data Engineer (PySpark & Python)
Data Engineer (PySpark & Python)

Highbrow Technology Inc • Dallas (TX)

On-site
USD 140,000 - 200,000
ETL Tester – PySpark
ETL Tester – PySpark

Robotics Technologies LLC • Boston (MA)

On-site
USD 90,000 - 120,000
Senior Data Engineer
Senior Data Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Atlanta (GA)

Hybrid
USD 93,000 - 133,000
ETL Tester – PySpark
ETL Tester – PySpark

Ethereum Technologies LLC • Boston (MA)

On-site
USD 85,000 - 120,000
Java Pyspark Developer
Java Pyspark Developer

Tata Consultancy Services Limited • Jersey City (NJ)

On-site
USD 110,000 - 135,000
Data Engineer
Data Engineer

INFOSYS NOVA HOLDINGS LLC • Charlotte (NC)

On-site
USD 90,000 - 135,000
ETL Tester – PySpark
ETL Tester – PySpark

Quantum Technologies LLC • Boston (MA)

On-site
USD 70,000 - 110,000