Data Engineer Cloudera

Tata Consultancy Services Limited; Irving (TX); On-site; USD 125,000 - 145,000; Full time

Job summary

Tata Consultancy Services Limited in Irving, TX is seeking a data engineer to design and maintain scalable ETL/ELT pipelines using PySpark, Spark SQL, and Hive. You will optimize data storage and partitions while handling large, diverse datasets.

The role requires strong Python skills, HiveQL/SQL proficiency, and experience with Parquet/ORC/Avro formats. Collaboration with data scientists and analysts is essential.

Qualifications

• Extensive Spark architecture knowledge, drivers, executors, and DAGs.
• Strong Python coding skills with PySpark for transformations.
• Proficiency in HiveQL/SQL and data partitioning techniques.
• Experience with optimized big data storage formats (Parquet/ORC/Avro).
• Solid data warehousing foundation, dimensional modeling, data lakes.
• Databricks Certification is a plus.

Responsibilities

• Design, build and maintain scalable ETL/ELT pipelines with PySpark/Spark SQL and Hive.
• Optimize data warehousing layouts, partitioning, and indexing for performance.
• Tune Spark jobs and manage data skew and memory usage.
• Ingest large, diverse datasets from relational DBs and flat files.
• Automate workflows using Airflow or platform schedulers.
• Collaborate with data scientists and analysts to deliver data solutions.

Skills

Apache Spark

PySpark API

HiveQL / ANSI SQL

Data Modeling (Star/Snowflake)

Data Warehousing Concepts

Databricks Certification

Education

Bachelor of Computer Science

Tools

Spark / Core

Apache Hive

Parquet/ORC/Avro

AWS EMR / Databricks

Cloudera Hadoop (Hadoop Admin)

Job description

Roles & Responsibilities:

• Data Pipeline Development & Maintenance: Design, build, and maintain highly scalable and efficient ETL/ELT data pipelines utilizing PySpark and Spark SQL , Hive for complex data transformations.
• Data Warehousing & Storage Optimization: Strategically manage data layout, partitioning, and indexing within Apache Hive and various cloud data lake solutions to optimize performance and accessibility.
• Performance Tuning & Optimization: Proactively identify and resolve performance bottlenecks in Spark jobs, leveraging Spark UI for in-depth analysis, effectively managing data skewness, and optimizing memory utilization.
• Diverse Data Integration: Develop robust solutions for ingesting high-volume and diverse datasets from both structured relational databases and unstructured flat files into our data ecosystem.
• Automated Workflow Orchestration: Implement and manage automated data workflows using industry-standard scheduling tools like Apache Airflow or platform-native schedulers, ensuring timely and reliable data delivery.
• Strategic Collaboration: Partner closely with data scientists, business analysts, and cross-functional enterprise teams to translate complex business requirements into technically sound and efficient data solutions.

Qualifications:

• Big Data Frameworks Expertise: Demonstrated high proficiency in Apache Spark architecture, including a deep understanding of drivers, executors, and Directed Acyclic Graphs (DAGs).
• Advanced Programming: Exceptional coding skills in Python and extensive experience with the PySpark API for developing intricate data transformations and processing logic.
• Querying & Schema Management: Strong command of HiveQL and ANSI SQL, coupled with expertise in data partitioning techniques and effective schema definition.
• Optimized Storage Formats: In-depth understanding and practical experience with optimized big data storage file formats such as Parquet, ORC, and Avro.
• Data Warehousing Fundamentals: Solid foundation in Dimensional Data Modeling, including Star and Snowflake schemas, and practical experience with Data Lakes concepts and implementation.
• Professional Certifications: Databricks Certified Associate Developer for Apache Spark

Preferred Qualifications

• CI/CD & DevOps Automation: Experience with Continuous Integration/Continuous Deployment (CI/CD) practices and automation tools like Git, Jenkins, or Ansible.
• Cloud Ecosystem Development: Experience in development experience utilizing cloud-native big data utilities (e.g., AWS EMR, AWS Databricks) within major cloud platforms.
• NoSQL Database Integration: Exposure to and experience with NoSQL databases such as HBase, Cassandra, or MongoDB.
• Professional Certifications: Databricks Certified

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE

Salary Range: $125,000 to $145,000 per year

Salary Range

$125,000 - $145,000

Location: Irving, TX

Function: TECHNOLOGY

Role: Engineer

ID: 436996

Skills: Cloudera Hadoop Administrator