Batch Automation Engineer: PySpark ETL & Ansible

Tata Consultancy Services; Irving (TX); On-site; USD 100,000 - 135,000; Full time

Benefits offered by this job

Restaurant d'entreprise

Indemnités de stage/alternance

Job summary

Tata Consultancy Services in Irving, TX seeks a data engineering professional to automate and optimize batch workflows with Shell scripting, manage secure file transfers with NDM and SFTP, and develop scalable PySpark ETL pipelines for large-scale data processing.

The role emphasizes automation via Ansible, infrastructure as code, and collaboration with cross-functional teams to ensure reliable, compliant data movement. Salary range $100,000-$135,000 annually; on-site position.

Qualifications

• Strong scripting and file-processing expertise for batch automation and workflow optimization.
• Experience developing scalable PySpark ETL pipelines for large-scale datasets.
• Experience managing secure enterprise file transfers using NDM and SFTP.
• Familiarity with infrastructure as code practices and Ansible.
• Collaborative work with cross-functional teams to ensure secure, reliable data movement.

Responsibilities

• Automate and optimize batch workflows using robust Shell scripting and file-processing techniques.
• Manage secure file transfers using NDM (Connect: Direct) and SFTP, including monitoring, troubleshooting, and operational support.
• Develop scalable PySpark ETL pipelines, ensuring high performance and reliability for large-scale data processing.
• Implement automation and configuration management using Ansible, following Infrastructure as best practices.
• Collaborate with cross-functional teams to ensure secure, reliable, and compliant data-movement operations.

Skills

Shell Scripting

PySpark

Operational Excellence

Education

Bachelor of Computer Science

Tools

NDM (Connect: Direct)

SFTP

Ansible

Job description

Tata Consultancy Services in Irving, TX seeks a data engineering professional to automate and optimize batch workflows with Shell scripting, manage secure file transfers with NDM and SFTP, and develop scalable PySpark ETL pipelines for large-scale data processing.

The role emphasizes automation via Ansible, infrastructure as code, and collaboration with cross-functional teams to ensure reliable, compliant data movement. Salary range $100,000-$135,000 annually; on-site position.