Staff - Data Engineer

alliancedata; Bengaluru; On-site; INR 3,000,000 - 5,500,000; Full time

Job summary

alliancedata in Bengaluru seeks a Staff Data Engineer to build and scale data ingestion pipelines, data warehouse/lake, and enterprise big data platform across multi-datacenter contexts. You will identify architecture, security, and automation needs, partnering with analytics teams.

You will mentor teams, implement streaming data with Spark/Hadoop, use CI/CD, and collaborate with data architects to deliver reliable analytics tools and dashboards for marketing and business users.

Qualifications

• Bachelor's degree in Computer Science or Information Systems or equivalent.
• Master's degree in Computer Science or STEM field or equivalent.
• 8–10 years of work experience in IT, SQL, Hadoop ecosystems, Data warehouse, data modeling.
• Experience in project management.

Responsibilities

• Demonstrate foundational knowledge of credit card operations, banking, loyalty rewards, retail, and regulations while working with the business.
• Guide on building data pipelines for multiple data sources and monitor data quality issues.
• Collaborate with data architects on data model management and version control.
• Extract, transform, and load data using Hadoop, Spark, SQL, and other big data tools.
• Develop automation to support analytics teams and contribute to PoCs and tool exploration.
• Ingest and process streaming data using Spark and HDFS; support dashboards and analytics tools.

Skills

SQL

Hadoop

Spark

Data modelling

ETL

Education

Bachelor's Degree in Computer Science or IS

Master's Degree in CS or STEM

Tools

Hive

Git

JIRA

Job description

Every career journey is personal. That's why we empower you with the tools and support to create your own success story.

Be challenged. Be heard. Be valued. Be you ... be here.

Job Summary

The Staff Data Engineer works on different projects of data engineering to support the use cases, data ingestion pipeline, data warehouse, data lake and enterprise big data platform with multi-datacenter contexts and identify potential process or data warehousing issues. Identify the architecture, infrastructure and interface to data sources, tools supporting automated data loads, security concerns, analytical models and data visualization. The Data Engineer team supports marketing analytic teams with analytical tools that enable our analytics and business communities to do their job easier, faster and smarter. The team brings together data from different internal & external partners and builds a curated Marketing analytics focused data & tools ecosystem. The Lead Data Engineer plays a crucial role in building this ecosystem depending on the Marketing analytics communities need.

Essential Job Functions

• Demonstrate foundational knowledge of credit card operations, banking, financial, loyalty rewards, retail, and credit card regulations while working with the business. Understand and employ common procedures and tools used in the Data Operations lifecycle (Agile Project Management, CI/CD, JIRA, etc.). Document important milestones and data lineage definitions. - (10%)
• Provides guidance on building data pipeline for various data sources, particularly from many external data sources. Works with other project team members to understand the use cases, familiar with data sources, provides recommendation on data ingestion processes, complete data ingestion, monitor and identify potential process or data quality issues. - (15%)
• Collaborate with Data Architects/Engineers on data model management and version control. Capture Critical Data Elements (CDE) and translate them into quality checks with Business/Data Engineers. Collaborate with business users, analysts, and architects to convert requirements into analytical tools and dashboards, aligning with usability best practices and design trends. - (10%)
• Demonstrates analytical, interpersonal and professional communication skills. Learns quickly and works effectively individually and as part of a team. Accesses, extracts, and transforms Credit and Retail data from a variety of sources of all sizes (including client marketing databases, 2nd and 3rd party data) using Hadoop, Spark, SQL, Big data technologies etc. - (10%)
• Provides automation help to analytical teams around data centric needs using orchestration tools, SQL and possibly other big data/cloud solutions for efficiency improvement. Supports Lead Data Engineer and Principal Data Engineer in new analytical proof of concepts and tool exploration projects. - (5%)
• Effectively manages time and computing resources in order to deliver on time/correctly on concurrent projects. Involved in creating POCs to ingest and process streaming data using Spark and HDFS. Answers and trouble shoots questions about data sets and analytical tools; Develops, maintains and enhances new and existing analytics tools/Frameworks to support internal customers/consumers. - (10%)
• Design, develop and implement data infrastructure and pipelines that collect, connect, centralize, and curate data from various internal and external data sources. Create automation systems and tools to configure, monitor, and orchestrate our data infrastructure and our data pipelines. - (10%)
• Ingests data from different sources, processes it according to the requirement document in order to store data to Hive or NoSQL database or different warehousing solutions. Involved in HDFS maintenance, and loading of structured and unstructured data. Evaluate new technologies for continuous improvements and collaborate closely with the product team to build out new data features. - (10%)
• Work with the data analysts and scientists to implement descriptive, forecasting, and predictive algorithms and models using the latest technologies. Make technology decisions for our data infrastructure. Applies knowledge in Agile Scrum methodology that leverages the Client BigData platform and used version control tool Git. - (10%)
• Perform as a Subject Matter Expert on various data domains and business processes. Understand and employ best practices in the field. Educate and mentor associates across the business in these best practices. Translate complex technical subjects for diverse audiences, ensuring alignment and understanding across technical and non-technical stakeholders. - (10%)

Minimum Qualifications

• Bachelor's Degree in Computer Science or Information Systems or equivalent experience.
• Master's Degree in Computer Science or STEM field or equivalent experience.
• 8-10 years of work experience required in IT, SQL, Hadoop Ecosystems, Data warehouse, Data Modelling.
• Experience on project management