Sr. Data Engineer ( big data )

Redolent Infotech Pvt. Ltd.; San Bruno (CA); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Market-rate contract

Job summary

Redolent Infotech Pvt. Ltd. in the San Francisco Bay Area is seeking a Sr. Data Engineer (Big Data) to design and build end-to-end data pipelines, with strong Java/Scala and Hadoop ecosystem skills.

You will work on real-time ingestion (Kafka), Teradata SQL, and large datasets, delivering robust, scalable solutions in a fast-paced environment.

Qualifications

• BS degree in computer science, math, or statistics preferred.
• 5+ years of experience in data engineering or related field; retail background a plus.
• Strong programming skills in Java or Python with end-to-end data pipelines.
• Teradata SQL performance optimization experience.
• Experience with real-time data ingestion (Kafka) and large datasets.

Responsibilities

• Design and build end-to-end data pipelines for large-scale data.
• Collaborate with teams to transform business requirements into data models and reports.
• Develop and optimize ETL processes and data transformations.
• Implement real-time data ingestion and batch processing using Hadoop ecosystem.
• Provide clear documentation and data lineage for downstream consumers.

Skills

Java

Scala

Flume

Python

Spark

MapReduce

YARN

Hadoop

Hive

Pig

Teradata SQL

Unix/Linux shell scripting

Kafka

Education

BS degree in Computer Science / Math / Statistics

Tools

Teradata Tools and Utilities (FastLoad, MultiLoad, BTEQ)

Cassandra

Automic Scheduler

R / SAS experience

Presto

Hbase

Tableau or BI tools

Job description

We have following urgent role with our DIRECT client

Title: Sr. Data Engineer ( Big Data)

Location: San Bruno, CA

Duration: 6+ Months

Rate: Market

Technical Requirements

• Development experience with Java, Scala, Flume, Python
• Development background with Spark or MapReduce/YARN is a must have
• This candidate is expected to have significant application development experience in the past either with Java or Python. We are looking for someone who was a Software Engineer in the past and turned into a Data Engineer and build end to end data pipelines from ground up
• Knowledge/experience on Teradata Physical Design and Implementation, Teradata SQL Performance Optimization
• Advanced SQL (preferably Teradata)
• Experience working with large data sets, experience working with distributed computing (MapReduce, Hadoop, Hive, Pig, Apache Spark, etc.)
• Strong Hadoop scripting skills to process petabytes of data
• Experience in Unix/Linux shell scripting or similar programming/scripting knowledge
• Experience in ETL/ processes
• Real time data ingestion (Kafka)

Nice to Have

• Experience with Teradata Tools and Utilities (FastLoad, MultiLoad, BTEQ, FastExport)
• Cassandra
• Automic scheduler
• R/R studio, SAS experience a plus
• Presto
• Hbase
• Tableau or similar reporting/dash boarding tool
• Modeling and Data Science background
• Retail industry background

Education

BS degree in specific technical fields like computer science, math, statistics preferred

Position Summary

• Very Strong engineering skills. Should have an analytical approach and have good programming skills.
• Provide business insights, while leveraging internal tools and systems, databases and industry data
• Minimum of 5+ years’ experience. Experience in retail business will be a plus.
• Excellent written and verbal communication skills for varied audiences on engineering subject matter
• Ability to document requirements, data lineage, subject matter in both business and technical terminology.
• Guide and learn from other team members.
• Demonstrated ability to transform business requirements to code, specific analytical reports and tools
• This role will involve coding, analytical modeling, root cause analysis, investigation, debugging, testing and collaboration with the business partners, product managers other engineering team.