Consultant- R01549620

Brillio; United States; On-site; USD 120,000 - 150,000; Full time

Job summary

A leading technology company is seeking a Lead Data Engineer to develop and maintain Big Data ETL jobs and work with technologies like Spark and AWS. The ideal candidate will have 5-10 years of experience, strong programming skills in Python or Scala, and an understanding of data security and compliance standards. This role offers an exciting opportunity to work on cutting-edge data processing pipelines.

Qualifications

• 5 to 10 years of experience in data engineering.
• Strong programming skills in Python, Java, or Scala.
• Experience with Big Data technologies and principles.

Responsibilities

• Develop and maintain Big Data ETL jobs.
• Conduct data analysis and profiling.
• Collaborate with cross-functional teams.

Skills

Python

Java

Scala

Spark

Hadoop

SQL

Databricks

Tools

Spark - Pyspark

HDFS

ETL

Data Warehousing

Hive

Job description

Lead Data Engineer

Primary Skills

• MapReduce, HDFS, Spark - Pyspark, ETL Fundamentals, SQL, SQL (Basic + Advanced), Spark - Scala, Python, Data Warehousing, Hive, Modern Data Platform Fundamentals, Data Modelling Fundamentals, PLSQL, T-SQL, Stored Procedures, Oozie

Job requirements

• Primary Skills (Must Have) : Programming language: Python / Java / Scala, Framework: Spark, Hadoop, Declarative language: SQL Secondary Skills (Good to Have) : Databricks , AWS Scope of Work :
• Developing and maintaining Big Data ETL jobs.
• Conducting data analysis and profiling.
• Collaborating with cross-functional teams (e.g., data scientists, business analysts) to align data engineering solutions with business objectives.
• Automating data workflows to increase system efficiency and reduce manual intervention.
• Monitoring, troubleshooting, and optimizing data pipelines to ensure high performance and reliability.
• Ensuring data security and compliance with Mastercard policies and industry standards.

• Role of Resource : Senior Data Engineer Role USP : Opportunity to work on cutting-edge Big Data technologies like Spark, Databricks, Cloud services with a focus on optimizing and scaling data processing pipelines.
• If Domain knowledge required : Banking / Mastercard data knowledge is an added advantage and preferable.

• Experience Range : 5 to 10 years