Azure Databricks Engineer - Assistant Manager
State Street
A complete application in a minute
Tailored resume and cover letter, ready to send.
Job summary
State Street is seeking an experienced Azure Data Engineer/Developer to design and build scalable data solutions on Microsoft Azure. You will work with Azure Databricks, Spark, Scala, Snowflake, and SQL to deliver robust ETL/ELT pipelines, data integration, and governance within Agile teams.
You will deploy and optimize data platforms, implement CI/CD pipelines with Azure DevOps, and ensure reliability and performance of data workloads in a modern cloud environment.
Qualifications
- Strong hands-on experience with Azure Databricks, Spark, Scala, Snowflake and SQL.
- Proven ability to design and implement data pipelines and data-warehousing solutions.
- Experience with CI/CD and DevOps in a data engineering context.
- Knowledge of data governance, security, and cloud-native data platforms.
Responsibilities
- Design and develop scalable, secure data engineering solutions on Azure.
- Build ETL/ELT pipelines and integrate Databricks with Snowflake and sources.
- Develop and optimize SQL queries, procedures, and transformation logic.
- Create reusable Scala/Spark components and manage streaming/batch data pipelines.
- Implement CI/CD pipelines and automate deployments across environments.
Skills
Education
Tools
Job description
We are seeking an experienced Azure Data Engineer/Developer with strong hands-on expertise in Azure Databricks, Apache Spark, Scala, Snowflake, SQL, and Azure DevOps.
The successful candidate will bring strong data-engineering expertise, experience with automated CI/CD pipelines and cloud migration, and the ability to deliver effectively within Agile teams.
Why This Role Is Important to Us
This role is central to building and modernizing scalable cloud-based data solutions. The engineer will design, develop, test, deploy, and support data platforms that improve processing efficiency, reliability, automation, and enterprise data delivery.
- Design and develop scalable, secure, and maintainable data-engineering solutions on Microsoft Azure.
- Build and support high-volume data-processing workloads using Azure Databricks, Apache Spark, and Scala.
- Develop reliable ETL/ELT pipelines for data ingestion, transformation, processing, and delivery.
- Design and implement Snowflake data-processing and data-warehouse solutions.
- Develop and optimize SQL queries, stored procedures, functions, and transformation logic.
- Integrate Databricks with Snowflake and other enterprise data sources.
- Create reusable, modular, and maintainable Scala and Spark components.
- Develop batch and streaming data pipelines using Lakeflow Connect, Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, Auto Loader, and Structured Streaming.
- Apply Unity Catalog capabilities for data governance, access control, lineage, catalog and schema management, and secure data sharing.
- Design and optimize Delta Lake solutions using ACID transactions, schema enforcement and evolution, time travel, change data feed, and table-maintenance practices.
- Build incremental and event-driven processing solutions using streaming tables, materialized views, Azure Event Hubs, or Kafka.
- Develop and tune Databricks SQL workloads using SQL warehouses, query profiling, Photon acceleration, and cost-optimization practices.
- Configure serverless computing, cluster policies, Databricks Runtime versions, workload isolation, and autoscaling based on workload requirements.
- Automate Databricks deployments using Declarative Automation Bundles, the Databricks CLI, REST APIs, service principals, and OAuth-based authentication.
- Monitor pipeline health, data quality, lineage, streaming backlogs, and resource usage using event logs, system tables, query history, and operational alerts.
- Implement data-quality controls, manage schema drift and quarantined records, and build resilient pipelines with checkpoint recovery, idempotent processing, and restart capabilities.
- Apply enterprise security practices, including managed identities, private networking, secure secret management, and Azure Key Vault integration.
- Support ML flow 3 capabilities for experiment tracking, model evaluation, observability, deployment, and model lifecycle management in Unity Catalog, where applicable.
- Tune and optimize Spark/Databricks, SQL, and Snowflake workloads.
- Troubleshoot applications, pipeline, deployment, and data-processing issues.
- Identify technical risks and production issues and drive them through resolution.
- Implement source-control, build, release, and deployment processes using Azure DevOps.
- Develop and maintain automated CI/CD pipelines for data-engineering applications.
- Support automated deployments across development, test, and production environments.
- Participate in application and data-workload migrations to Azure and other modern cloud platforms.
- Collaborate with engineering, infrastructure, and operations teams to deliver solutions successfully.
- Contribute to solution design, technical reviews, code reviews, and release activities.
- Work within Agile, Scrum, or Kanban delivery models and manage deliverables against committed schedules.
- Strong hands-on experience with Azure Databricks, Apache Spark, Scala, Snowflake, and SQL.
- Proven SQL development and query-optimization skills.
- Solid understanding of data warehousing and ETL/ELT concepts.
- Experience with large-scale data processing and transformation.
- Experience with Microsoft Azure, including Azure Databricks and Azure data-integration services.
- Experience migrating cloud applications and data workloads.
- Hands-on experience with Azure DevOps and Git-based source control.
- Ability to design and implement CI/CD pipelines.
- Experience automating build and deployment processes.
- Experience managing environment-based deployments and releases.
- Data-pipeline design and development.
- Performance tuning and optimization.
- Debugging, root-cause analysis, and production issue resolution.
- Code reviews and adherence to engineering standards.
- Experience with Agile development methodologies.
- Azure Data Factory (ADF).
- Python.
- Azure SQL or Azure SQL Managed Instance.
- Oracle or SQL Server.
- Jira and Confluence.
- Experience migrating legacy or on-premises workloads to Azure.
- Financial services technology experience.
- Azure Developer Associate, Azure Data Engineer, or Snowflake certification.
- Six or more years of overall IT or software-engineering experience preferred.
- Strong practical experience developing enterprise-scale data solutions.
- Demonstrated hands-on experience with Databricks/Spark, Scala, Snowflake, and SQL.
- Experience with enterprise CI/CD and DevOps practices.
- Bachelor’s degree in computer science, Engineering, Information Technology, or a related discipline preferred.
- Strong analytical, problem-solving, troubleshooting, and root-cause-analysis skills.
- Ability to independently own technical deliverables and meet committed timelines.
- Ability to collaborate effectively across engineering, infrastructure, and operations teams.
- Clear written and verbal communication skills.
- Ability to translate business and data requirements into scalable technical solutions.
- Strong focus on code quality, automation, reliability, and maintainability.
Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.
We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you'll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.
As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.
Discover more information on jobs at StateStreet.com/careers
Read our CEO Statement
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
