Senior Data Engineer

United Network For Organ Sharing; Richmond (VA); On-site; USD 140,000 - 190,000; Full time

Job summary

United Network For Organ Sharing is seeking a Senior Data Engineer to lead end-to-end Azure-based data solutions for analytics, reporting, and operations. You will design scalable pipelines, build a Data Lakehouse, and mentor junior engineers while collaborating with data scientists and business stakeholders.

Key duties include implementing secure data pipelines with Azure Data Factory, Databricks, and ADLS Gen2, ensuring data quality, and driving modernization across teams in a fast-paced cloud

Qualifications

• 8+ years in data engineering with enterprise-scale data solutions.
• 2+ years building Azure cloud-native data platforms.
• Strong SQL, Python, and Spark experience in a data lakehouse context.
• Experience with ETL/ELT, data quality, and monitoring frameworks.
• Ability to communicate with data scientists, analysts, and business stakeholders.

Responsibilities

• Architect and implement secure, scalable data pipelines in Azure.
• Lead ETL/ELT design using Azure Data Factory and Databricks.
• Build and maintain Data Lakehouse architectures and Delta Lake structures.
• Ensure data quality, validation, and monitoring across pipelines.
• Mentor junior engineers and lead code reviews for best practices.
• Collaborate with stakeholders to translate requirements into data solutions.
• Support CI/CD for data pipelines using Azure DevOps and Git.

Skills

MS SQL Server

Python (pandas, PySpark)

Azure Data Factory

Azure Functions

Azure Data Lake Storage

Azure Databricks

Spark/Delta Lake

Data Lakehouse

REST APIs

Git / CI/CD

Azure Synapse Analytics

Data governance

Big Data technologies

Education

4-year degree in CS/Engineering or related IT field

Tools

Azure DevOps

Git

Databricks Repos

Azure Synapse

Job description

Position DescriptionWe are seeking a Senior Data Engineer to lead the design and implementation of end-to-end data solutions within a modern Azure-based cloud environment. This hands-on, technical role will partner closely with data scientists, analysts, and business stakeholders to ensure that data is accurate, accessible, and optimized for analytics, reporting, and operational needs. You will play a key role in building Data Lakehouse, scalable data pipelines and integrations while championing best practices, mentoring junior engineers, and leading strategic data initiatives across teams.Key ResponsibilitiesArchitect and implement secure, scalable data pipelines using Azure Data Factory, Azure Functions, and Azure Data Lake StorageDesign, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks (PySpark/Spark SQL) to ingest, transform, and process large volumes of structured and unstructured dataBuild and manage data pipelines using Azure services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure Event Hubs, and Azure SQL DatabaseImplement data quality, validation, and monitoring frameworks to ensure reliability and accuracy of data pipelinesCollaborate cross-functionally with data scientists, analysts, and business stakeholders to understand data requirements and deliver fit-for-purpose data solutionsLead modernization and optimization efforts, improving pipeline performance, maintainability, and scalabilitySupport CI/CD practices for data pipelines using tools such as Azure DevOps, Git, and Databricks ReposDetect and resolve data quality issues; implement automated audits and monitoring processesTroubleshoot and resolve production data pipeline issues, ensuring high availability and minimal downtimeAct as a technical leader on schema design, performance tuning, and Azure data architectureMentor and support junior engineers across data engineering, analytics, and BI teamsParticipate in and lead code reviews, promoting clean, well-documented, and testable codeStay current on trends in data engineering and cloud technologies, identifying opportunities to innovateMinimum Requirements8+ years of hands-on experience in data engineering, including designing and implementing enterprise-scale data solutions.2+ years of experience developing and operating Azure cloud-native data platforms.Critical SkillsExpertise in MS SQL Server, Python (pandas, PySpark), Azure Data Factory, Azure Functions and Azure Data Lake Storage.Strong expertise with Azure Databricks, including Spark (PySpark/Scala), Delta Lake, and cluster/job optimization.Solid understanding and hands-on experience building Data Lakehouse architectureExperience working with a variety of file formats (e.g., CSV, JSON, XML, Parquet).Experience with version control (Git) and CI/CD pipelines for data engineering workflowsFamiliarity using REST APIs for data extraction and integration.Proven experience designing and implementing data solutions.Strong understanding of cloud architecture, data warehousing and modern data stack components.Additional Skills & QualificationsDemonstrated ability to perform root cause analysis on data and processing issuesStrong problem-solving skills with the ability to explain technical concepts to non-technical audiencesA successful history of manipulating, processing and extracting value from large disparate datasetsExperience with Big Data technologies such as Databricks, Spark, or Azure SynapseKnowledge of CI/CD workflows, version control, and agile development practicesFamiliarity with data governance, privacy, and compliance frameworksExperience with data warehousing, analytics tools, and BI platformsFamiliarity with streaming data technologies (Azure Event Hubs, Kafka, Structured Streaming)Education4-year degree in computer science, engineering or other related IT field of study, or equivalent professional work experiencePhysical RequirementsGeneral office demandsProlonged periods of sitting at a desk and working on a computer.Frequent reaching, handling, and fine manipulation for using office equipment, filing, and managing paperwork.Manual dexterity sufficient to operate a keyboard, mouse, and other office tools.Occasional standing, walking, and bending.Ability to lift up to 10-20 pounds occasionally.Vision abilities required include close vision for computer work and reading documents.Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.