Senior Data Engineering Specialist

U.S. Bank; Irving (TX); Hybrid; USD 105,400 - 124,000; Full time

Benefits offered by this job

Healthcare benefits

401(k) and retirement plans

Paid vacation

Parental leave

Job summary

A leading financial institution is seeking an experienced Data Engineer to manage data-centric solutions in a cloud environment. The role involves designing scalable data architectures, optimizing data processes, and ensuring data governance and security. Candidates should have a strong background in Data Engineering technologies and extensive experience with Azure cloud services. This position offers a hybrid work schedule, emphasizing collaboration and innovation.

Qualifications

• 10+ years of hands-on experience in Data engineering, Cloud platform management, and performance optimization.
• Hands-on experience with Hadoop, Hive, Spark, and migration of Big Data into Azure cloud services.
• Working knowledge of Data Integration and Data Lake Architectures.

Responsibilities

• Guide team to migrate from on-prem Cloudera to Azure cloud environment.
• Design and implement scalable data lake solutions.
• Manage data governance, quality, and security across cloud environments.

Skills

Data Engineering

Cloud Management

Hadoop

Spark

Hive

Python

Linux

Ansible

Documentation

Collaboration

Education

Advanced degree in Computer Science, Engineering or related field

Tools

Databricks

Snowflake

Azure Cloud

HashiCorp Vault

CyberArk

Cloudera

Job description

At U.S. Bank, we’re on a journey to do our best. Helping the customers and businesses we serve to make better and smarter financial decisions and enabling the communities we support to grow and succeed. We believe it takes all of us to bring our shared ambition to life, and each person is unique in their potential. A career with U.S. Bank gives you a wide, ever-growing range of opportunities to discover what makes you thrive at every stage of your career. Try new things, learn new skills and discover what you excel at—all from Day One.

Job Description

• Data Lake Architecture and Experience:
  • Guide team to migrate from on‑prem Cloudera to Azure cloud environment.
  • Design and implement scalable data lake solutions using Snowflake and Databricks, and develop and optimize data pipelines for ingestion, transformation, and storage.
  • Manage data governance, quality, and security across cloud environments; implement performance tuning, automation, and CI/CD for data workflows.
  • Collaborate with cross‑functional teams to support cloud migration activities.
• Cloudera Cluster Management:
  • Install, configure, manage, and monitor Cloudera Hadoop clusters, ensuring high availability, performance, and security. Manage HDFS, YARN, and other ecosystem components.
• Performance Optimization:
  • Tune Hadoop, Hive, and Spark jobs and configurations for optimal performance, efficiency, and resource utilization. Optimize queries, manage partitions, and leverage in‑memory capabilities.
• Troubleshooting and Support:
  • Diagnose and resolve issues related to Linux servers, networks, cluster health, job failures, and performance bottlenecks. Provide on‑call support and collaborate with other teams to ensure smooth operations.
• Security, Governance, and Secrets Management:
  • Implement and manage security measures within the Cloudera environment, including Kerberos, Apache Ranger, and Atlas, to ensure data governance and compliance.
  • Set up and manage HashiCorp Vault for secure keys and secrets management.
  • Utilize CyberArk for privileged access management and secure administrative tasks on the cluster.
• Data and Application Migration:
  • Migrate Hadoop, Hive, and Spark data and applications to Azure cloud services such as Azure Synapse Analytics, Azure Databricks, or Snowflake. Ensure data integrity, performance tuning, and validation.
  • Develop scripts (e.g., shell, Ansible, Python) to automate administrative tasks, deployments, and monitoring. Work with users to develop, debug, optimize Hive/Spark/Python programs that connect to the Cloudera environment.
• Documentation:
  • Create and maintain documentation for system configurations, operational procedures, and troubleshooting knowledge bases.
• Vendor Collaboration:
  • Work closely with the Cloudera vendor to stay current with the latest releases, perform upgrades, and address vulnerabilities.

Basic Qualifications

• Advanced degree in Computer Science, Engineering, or related field.
• Deep expertise in Data Engineering and Management technologies, synthetic data, automation, advanced analytics.
• 10+ years of hands‑on experience in Data engineering, Cloud platform management, and performance optimization.
• Hands‑on experience with Hadoop, Hive, Spark, and migration of Big Data into Azure cloud services.
• Working knowledge and hands‑on experience in Data Integration and Data Lake Architectures with Databricks and Snowflake platforms.
• Working knowledge of Microsoft Azure cloud and big data migration to cloud platforms.
• Experience with HashiCorp Vault for secrets and keys management.
• Experience with CyberArk for privileged access and administrative security tasks.
• Proficiency in Linux, clustering, and distributed systems.
• Expertise in Hive and Spark for data processing and analytics.
• Expertise in Hadoop ecosystem components such as HDFS, YARN, Hive, Spark, and Sqoop.
• Proficiency in languages such as shell, Ansible, C/C++, Java, and Python, PySpark for automating workflows, deployments, and monitoring.
• Expertise in Linux, Network, Python scripting, DNS, Kerberos, LDAP/AD, MySQL, PostgreSQL, JupyterHub.
• Experience in creating and maintaining documentation for system configurations, operational procedures, and troubleshooting knowledge bases.
• Strong problem‑solving skills and the ability to diagnose and resolve system failures and performance bottlenecks.
• Excellent communication and collaboration skills to work effectively with cross‑functional teams.

Preferred Skills and Experience

• Strong experience with Cloudera Hadoop distribution administration.
• Strong problem‑solving, analytical skills.
• Deep understanding of banking, operations, risk, and compliance—pairing technical fluency with business and regulatory acumen.
• Experience on change management process and addressing PCI and PII data requirements.
• Experience in cloud‑native architectures (Azure/AWS), Kubernetes, Docker, and containerized workflows.
• Experience in implementing secure systems with secure API design, IAM integration, Kerberos, Active Directory, and Azure SSO.
• The role offers a hybrid/flexible schedule, which means there's an in‑office expectation of 3 or more days per week and the flexibility to work outside the office location for the other days.

Benefits

• Healthcare (medical, dental, vision)
• Basic term and optional term life insurance
• Short‑term and long‑term disability
• Pregnancy disability and parental leave
• 401(k) and employer‑funded retirement plan
• Paid vacation (from two to five weeks depending on salary grade and tenure)
• Up to 11 paid holiday opportunities
• Adoption assistance
• Sick and Safe Leave accruals of one hour for every 30 worked, up to 80 hours per calendar year unless otherwise provided by law

U.S. Bank is an equal opportunity employer. We consider all qualified applicants without regard to race, religion, color, sex, national origin, age, sexual orientation, gender identity, disability or veteran status, and other factors protected under applicable law.

If there’s anything we can do to accommodate a disability during any portion of the application or hiring process, please refer to our disability accommodations for applicants.

U.S. Bank participates in the U.S. Department of Homeland Security E‑Verify program in all facilities located in the United States and certain U.S. territories. The E‑Verify program is an Internet‑based employment eligibility verification system operated by the U.S. Citizenship and Immigration Services.

Pay Range: $105,400.00 - $124,000.00

Posting may be closed earlier due to high volume of applicants.