Senior Data Engineer: AI-Powered Data Platform on Azure

HubQuest; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

Aether Biomedical is seeking a Senior Data Engineer to own and deliver complex data pipelines on Azure and Databricks. You will write production-grade Python and PySpark code, drive end-to-end solutions, and collaborate across Data Management to modernize data platforms.

The role emphasizes independent problem solving, architectural input, and hands-on coding with exposure to AI-enabled data discovery and governance capabilities.

Qualifications

• 8+ years of professional experience in Data Engineering, Software Engineering, or a closely related field.
• Strong, hands-on commercial experience with Azure Databricks.
• Very good knowledge of the Azure cloud ecosystem and cloud-based data architectures.
• Excellent programming skills in Python; strong coder, not just notebook usage.
• Advanced PySpark skills and experience with large-scale data processing.
• Advanced SQL with complex transformations, optimization, and analytics.
• Experience designing and delivering solutions independently with limited supervision.
• Ability to understand unfamiliar problems and drive implementation.
• Strong troubleshooting and problem-solving skills.
• Experience with production-grade engineering practices (version control, testing, CI/CD, monitoring, deployment).
• Strong understanding of software development principles and maintainable code.
• High ownership, independence, and proactivity.
• Excellent English communication skills.

Responsibilities

• Design, develop, and optimize complex data solutions and data pipelines on Azure and Databricks.
• Take end-to-end technical ownership from understanding the challenge to production deployment.
• Write production-grade code using Python, PySpark, and SQL.
• Work with Azure Databricks and Databricks ecosystem features.
• Explore AI-enabled capabilities within the data platform, including Databricks AI, RAG, data discovery, and conversational access to enterprise data.
• Contribute to migration and modernization of data platforms into Databricks-based architectures.
• Work with Unity Catalog and scalable data discovery, governance, and access approaches.
• Solve technically difficult problems with minimal predefined guidance.
• Identify opportunities to improve architecture, performance, scalability, reliability, and maintainability.
• Collaborate with Data Engineers, Architects, Product Owners, DataOps, QA, DevOps, and other tech teams.
• Support architectural discussions and contribute to technical design decisions across Data Management.

Skills

Azure Databricks

Python

PySpark

SQL

Data engineering

End-to-end ownership

Cloud architectures

Tools

Databricks

Job description

Aether Biomedical is seeking a Senior Data Engineer to own and deliver complex data pipelines on Azure and Databricks. You will write production-grade Python and PySpark code, drive end-to-end solutions, and collaborate across Data Management to modernize data platforms.

The role emphasizes independent problem solving, architectural input, and hands-on coding with exposure to AI-enabled data discovery and governance capabilities.