Sr. Data Engineer-Hadoop,Hive,Spark,Trino, Star Rocks, Clickhouse

Visa Consolidated Support Services India; Bengaluru; Hybride; INR 2 500 000 - 4 500 000; Plein temps

Résumé du poste

Visa Bengaluru seeks a Sr. Data Engineer to collaborate with product, business and security teams to design data-driven features. You will own data pipelines, ensure data quality, and implement scalable architectures using Hadoop, Spark and modern querying tools.

You will build automation, participate in CI/CD, and apply secure coding practices while integrating AI/ML services where applicable. On-site with onboarding expectations and hybrid options.

Qualifications

• 3+ years of relevant work experience or 5+ years with degree.
• Experience solving data problems with data technologies.
• Experience with distributed data systems (Hadoop, Hive, HBase, Spark, Trino).
• Experience designing data pipelines and large-scale processing jobs.
• Proficient in Java or Python, SQL, Git; production troubleshooting.
• Experience with CI/CD, containerization and orchestration.

Responsabilités

• Collaborate with stakeholders to clarify requirements and identify product stakeholders.
• Translate requirements into architecture designs and communicate to engineers.
• Implement maintainable, reusable code and participate in reviews.
• Own data products/pipelines end-to-end, manage technical debt.
• Develop tests and security scans for moderately complex use cases.
• Build tools and automation for batch and real-time data processing.
• Write queries and automate data extraction across pipelines.
• Ensure data quality and governance across features.
• Document troubleshooting procedures for advanced problems.
• Manage upgrades and patches for security remediation.

Connaissances

Data technologies

Hadoop

Hive

HBase

Spark

Trino

Data pipelines

ETL/ELT

Java

Python

SQL

Git

CI/CD

Docker

Kubernetes

Observability tools

AI/ML services

Secure coding

Regulatory standards

Formation

Bachelor's degree

Advanced degree

Outils

Hadoop

Hive

HBase

Spark

Trino

Description du poste

About Us

Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.

At Visa, you'll have the opportunity to create impact at scale tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.

Join Visa and do work that matters to you, to your community, and to the world. Progress starts with you.

Job Description

Role Summary:
Sr. Data Engineers are responsible for collaborating with stakeholders, including product teams, business partners, and cybersecurity, to identify and clarify requirements for data-driven features and products. They independently understand the data ecosystem, security, privacy, and retention requirements necessary to support business and product features. Sr. Data Engineers create and document technical designs for components, identify systems, architectures, tools, and storage needs, and present technical solutions to peers. They stay current with market trends and industry dynamics, applying this knowledge to drive innovation and ensure the safety and security of product components.
All roles require digital fluency, including the ability to work with emerging technologies such as Generative AI tools (e.g. ChatGPT, Microsoft Copilot) to support everyday work.

• Collaborate proactively with stakeholders to clarify requirements and identify key product stakeholders.
• Translate functional requirements into architecture designs for one or more components, leveraging existing architecture design patterns, and communicate these to Data Engineers.
• Implement extensible, maintainable, and reusable code using appropriate coding patterns, guidelines, and best practices; participate in code reviews to ensure standards are followed.
• Provide end-to-end ownership of data products or data pipelines, including mitigation of technical debt and documentation of errors or unexpected bugs.
• Identify and conduct required testing for moderately complex use cases, including unit testing, coding standards, and security scans.
• Build tools and standard automation processes to assist in transforming, managing, accessing, deploying, and monitoring data processes in batches and in real time.
• Write queries to extract and compile raw data across end-to-end pipelines, and implement orchestration techniques to automate data extraction logic.
• Ensure adherence to data management principles, governance, and tools to maintain data quality across multiple features.
• Develop and update troubleshooting guidance and procedures for reviewing, addressing, and fixing advanced problems flagged by testing.
• Manage software upgrades and server patches for security remediation where applicable.

Visa requires at least 3 days in office, expectations of these days will be confirmed by your Hiring Manager.

Qualifications

Basic Qualifications:

• 3+ years of relevant work experience and a Bachelor's degree, OR 5+ years of relevant work experience.
• Experience in solving data problems using data technologies
• Experience with Hadoop, Hive, HBase, Spark, Trino, or comparable distributed data systems.
• Experience in designing and implementing data pipelines and large-scale data processing jobs.
• Experience in writing, modifying, and reviewing high-quality, testable, and efficient code.
• Data pipelines, ETL/ELT, data-quality frameworks, Java or Python, SQL, Git, and production troubleshooting.
• Experience in developing unit, integration, and end-to-end tests with a focus on automation.
• Experience in building and maintaining CI/CD pipelines and deploying applications using containerization and orchestration tools.
• Experience in monitoring applications in production and troubleshooting issues using observability tools.
• Experience in integrating AI or ML services into applications, where applicable.
• Experience in collecting and analyzing metrics to guide optimizations and improvements.
• Experience in partnering with stakeholders to clarify requirements and ensure deliverables meet business needs.
• Experience in applying secure coding practices and complying with regulatory standards.

Preferred Qualifications:

• 3 or more years of work experience with a bachelors degree or more than 2 years of work experience with an Advanced Degree (e.g. Masters, MBA, JD, MD).
• Experience in solving data problems using data technologies
• Experience with Hadoop, Hive, HBase, Spark, Trino, or comparable distributed data systems.
• Experience in contributing in Open-source components.
• Use/Build AI agents, and agentic patterns to improve development, testing, operations, troubleshooting, automation, onboarding, and documentation.
• Data pipelines, ETL/ELT, data-quality frameworks, Java or Python, SQL, Git, and production troubleshooting.
• Experience in enterprise-class systems engineered for high availability, low latency, scalability, and supportability. Architecture depth across storage/compute separation, query and table formats, streaming, metadata, governance, and interoperability.
• Experience in building and pushing code into production.
• Experience in implementing and supporting real-time systems.

Visa is an EEO Employer

Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.