Lead Azure Data Engineer

Ryan Specialty, LLC

Carmel (IN)

On-site

USD 140,000 - 175,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Excellent benefits
Paid time off

Job summary

Ryan Specialty, LLC is seeking a Lead Azure Data Engineer to drive data architecture and scalable pipelines on the Azure cloud. You will mentor engineers, implement real-time and batch data integration, and ensure data quality across sources using Delta Parquet and Databricks.

The role emphasizes governance with Azure Purview, Profisee MDM integration, and strong Tableau BI capabilities. Requires extensive SQL Server experience and leadership in a fast-paced environment.

Qualifications

  • Bachelor’s degree required; Master’s degree preferred.
  • 12+ years of data engineering experience; or 8+ years with a Master’s degree.
  • Subject matter expert in SQL Server, Azure Data Factory, Databricks, Delta Parquet, and SSIS/SSRS.
  • Proficiency with Profisee MDM integration, CDC, and data governance using Azure Purview.
  • Experience with Tableau BI reporting and automation in Azure cloud.
  • Licenses & Certifications such as ADAC, SBAPC, or CCP Data Engineer a plus.

Responsibilities

  • Provide technical leadership in data architecture and engineering.
  • Mentor junior data engineers and team members.
  • Explore and implement new data engineering technologies and practices.
  • Manage and enhance SSIS packages and SSRS reports for BI.
  • Design and maintain scalable data pipelines using Azure Data Factory.
  • Develop and optimize Databricks notebooks for big data processing.
  • Implement Delta Parquet storage in Azure Data Lake Storage.
  • Set up Change Data Capture and data cataloging with Azure Purview.
  • Enhance Tableau reporting and integrate with Azure data services.
  • Drive CI/CD pipelines using Azure DevOps and automate processes.

Skills

Big Data Analytics
Leadership
Data Governance
Data Modeling
Data Processing

Education

Bachelor’s degree
Master’s degree

Tools

SSIS
SSRS
Azure Data Factory
Databricks
Delta Parquet
Profisee MDM
Azure Purview
Change Data Capture
Tableau
Azure DevOps

Job description

Position Summary

The Lead Azure Data Engineer will be reporting to VP of Data Engineering and will be responsible for implementing data solutions and collaborating with Business Analyst, Data Analyst, IT Teams and key stake holders. This role will lead projects and coach junior staff on build and optimize data pipelines, integrating data from various sources, ETL (Extract, Transform, Load) processes and ensuring data quality and consistency. The Lead will ensure that our data engineering practices are aligned with the latest technologies and industry best practices, focusing on performance, reliability, and efficiency.

What will your job entail?
Job Responsibilities:
  • Provide technical leadership in data architecture and engineering, collaborating with solution stakeholders to align data solutions with business objectives.
  • Mentor less experienced data engineers and team members, fostering a collaborative and innovative work environment.
  • Explore and implement new technologies and practices in data engineering to continually improve our data processes and infrastructure.
  • Manage and enhance complex SSIS packages for data integration and ETL processes, along with SSRS reports for business intelligence.
  • Design, build, and maintain scalable and efficient data pipelines using Azure Data Factory, implementing both batch and real-time data integration processes.
  • Develop and optimize Databricks notebooks for big data processing and analytics in Azure Databricks environments.
  • Implement and manage data storage solutions using Delta Parquet files in Azure Data Lake Storage, optimizing for query performance and cost-efficiency.
  • Integrate and configure Profisee Master Data Management tools within Azure, focusing on data governance and quality.
  • Set up and manage Change Data Capture processes for real-time data synchronization in Azure data platforms.
  • Implement data cataloging, classification, and lineage using Azure Purview, developing data governance policies, and ensuring compliance.
  • Enhance and support Tableau reporting solutions, developing complex dashboards, and integrating with Azure data services.
  • Drive automation and process optimization using Azure services and scripting and implement CI/CD pipelines using Azure DevOps.
  • Lead SQL Server database optimization and performance tuning efforts.
  • Lead, Coach and Mentor junior staff on projects.
Work Experience and Education:
  • Bachelor’s degree required, Master’s degree preferred; Computer science, Information Technology or Engineering is preferred. Any other related discipline or commensurate work experience considered.
  • 12+ years of experience in Data Engineering with a Bachelor’s degree, or 8 years of experience with a Master’s degree is preferred.
  • Subject Matter Expert in SQL Server Management, Azure Data Factory Development, Databricks Engineering, Delta Parquet File Management, and SQL Server Integration Services.
  • Proficiency in Profisee MDM Integration, Change Data Capture implementation, and data quality and governance with Azure Purview.
  • Proven experience in Tableau and BI Reporting, and strong skills in Automation and Process Licenses & Certifications:
Licenses & Certifications:
  • Advanced Data Analytics Certifications (ADAC), SAS Statistical Business Analyst Professional Certification (SBAPC), or Cloudera Certified Professional (CCP) Data Engineer a plus.
Skills:
Technical/Functional Skills:
  • Big Data Analytics
  • Leadership
  • Data Governance
  • Data Modeling
  • Data Processing
  • Data Warehousing
  • Database Management
  • Knowledge of ETL processes
  • Machine Learning
  • Predictive Analysis
  • Programming Languages
Behavioral Skills:
  • Critical Thinker
  • Communication
  • Relationship Building
  • Effective Communication
  • Time Management
  • Problem Solving
  • Attention to Detail

Applicants must be authorized to work for any employer in the U.S. We are unable to sponsor or take over sponsorship of an employment visa at this time.

Ryan Specialty is an Equal Opportunity Employer. We are committed to building and sustaining a diverse workforce throughout the organization. Our vision is an inclusive and equitable workplace where all employees are valued for and evaluated on their performance and contributions. Differences in race, creed, color, religious beliefs, physical or mental capabilities, gender identity or expression, sexual orientation and many other characteristics bring together varied perspectives and add value to the service we provide our clients, trading partners and communities. This policy extends to all aspects of our employment practices, including but not limited to, recruiting, hiring, discipline, firing, promoting, transferring, compensation, benefits, training, leaves of absence and other terms, conditions and benefits of employment.

How We Support Our Teammates

Ryan Specialty seeks to offer our employees a comprehensive and best-in-class benefits package that helps them — and their family members — achieve their physical, financial, and emotional well-being goals. In addition to paid time off for company holidays, vacation, sick and personal days, Ryan offers paid parental leave, mental health services and more.

The target salary range for this position is $140,000.00 - $175,000.00 annually. The wage range for this role considers many factors, such as training, transferable skills, work experience, licensure and certification, business needs, and market demands. The pay range is subject to change and may be modified in the future. Full-time roles are eligible for bonuses and benefits. For additional information on Ryan Specialty Total Rewards, visit our website https://benefits.ryansg.com/.

We provide individuals with disabilities reasonable accommodations to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment in accordance with applicable law. Please contact us to request an accommodation at HR@Ryansg.com.

Ryan Specialty, LLC and its affiliates (“Ryan Specialty”) do not accept unsolicited resumes or candidate profiles submitted by any staffing agencies, placement services or professional recruiters (“Agencies”), including any submissions made through our website or to email accounts of Ryan Specialty employees. Agencies must be registered in Ryan Specialty’s agency portal and have a fully executed contract for service between the Agency. Agency fee agreements will only be valid if provided in such fully executed contract between the Agency and the Agency.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Azure Data Engineer
Lead Azure Data Engineer

RT Specialty, LLC • Carmel (IN)

On-site
USD 140,000 - 175,000
Full Stack Development Lead
Full Stack Development Lead

Ryan Specialty, LLC • Carmel (IN)

On-site
USD 128,000 - 160,000
Bonuses
Comprehensive benefits package
Disability accommodations
Business Systems Analyst (Contract)
Business Systems Analyst (Contract)

Ryan Specialty, LLC • Chicago (IL)

On-site
USD 75,000 - 110,000
Full Stack Development Lead
Full Stack Development Lead

Ryan Specialty • Chicago (IL)

On-site
USD 128,000 - 160,000
Business Solutions Director
Business Solutions Director

Ryan Specialty, LLC • Chicago (IL)

On-site
USD 142,000 - 177,000
Bonuses
Comprehensive benefits package
Paid time off
+2
Business Systems Analyst
Business Systems Analyst

Ryan Specialty, LLC • Chicago (IL)

On-site
USD 88,000 - 110,000
Bonuses
Benefits
Business Systems Analyst
Business Systems Analyst

Ryan Specialty Group • Cockeysville (MD)

On-site
USD 88,000 - 110,000
Accounts Payable Manager
Accounts Payable Manager

Ryan Specialty • Chicago (IL)

On-site
USD 85,000 - 100,000
Inside Broker, ProExec
Inside Broker, ProExec

Ryan Specialty • Chicago (IL)

On-site
USD 146,000 - 183,000
Bonuses
Benefits package
Property & Casualty Underwriting Manager
Property & Casualty Underwriting Manager

Ryan Specialty, LLC • Chicago (IL)

On-site
USD 120,000 - 150,000
Bonuses
Benefits package