Engineer

Tata Consultancy Services; Dallas (TX); On-site; USD 100,000 - 105,000; Full time

Job summary

Tata Consultancy Services is looking for a Data Engineer to design, develop, and maintain data pipelines using PySpark in a Big Data environment. You will ensure data quality and implement ETL processes to migrate data across systems.

The ideal candidate has over 8 years of experience in Hadoop and PySpark development, along with expertise in cloud platforms like AWS and Databricks. Strong problem-solving skills and a Bachelor's in Computer Science are required.

The position offers a competitive salary range of $100,000 to $105,000 annually.

Qualifications

• 8+ years of professional experience in Hadoop, PySpark/Python development.
• Proven expertise in PySpark and handling large volumes of data.
• 3+ years of experience in AWS, Databricks/Snowflake, Airflow.

Responsibilities

• Design, develop, and maintain data pipelines using PySpark.
• Implement ETL processes for data migration across systems.
• Collaborate with teams to identify and resolve data-related issues.

Skills

Hadoop

PySpark/Python

AWS

Databricks

Airflow

CI/CD

Debugging

Communication

Education

Bachelor of Computer Science

Tools

Git

Docker

Job description

Job Description

As a Data Engineer, you will be responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing in a Big Data environment using predominantly PySpark/Python. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems using PySpark.

Responsibilities

• Design, develop, and maintain robust, scalable high-performance data pipelines using PySpark.
• Create data pipelines, ensuring data quality, and implement ETL processes to migrate and deploy data across systems.
• Migrate Ab Initio ETL applications into PySpark-based data pipelines.
• Migrate on‑premises workloads into cloud platforms (AWS, Databricks, Snowflake) based on use cases.
• Collaborate with cross-functional teams to identify and resolve data-related issues.
• Stay updated with the latest advancements in data engineering and integrate innovative approaches for sustained competitive advantage.

Qualifications

• 8+ years of professional experience in Hadoop, PySpark/Python development.
• Proven expertise in PySpark and handling large volumes of data.
• 3+ years of experience in AWS, Databricks/Snowflake, Airflow.
• Experience with CI/CD pipelines and version control systems (e.g., Git).
• Strong debugging and problem‑solving skills.
• Excellent communication and collaboration skills.

Good to Have Skills

• AWS EKS experience, Docker, and containers.

Salary Range

$100,000 to $105,000 per year

Education

Bachelor of Computer Science