Cloud Data Engineer: Azure Databricks & AI Pipelines

AmTrust Financial Services, Inc.; Cleveland (OH); On-site; USD 110,000 - 150,000; Full time

Benefits offered by this job

Medical & Dental Plans

Life Insurance

Health Care Flexible Spending

Dependent Care

401k Savings Plans

Paid Time Off

Job summary

AmTrust Financial Services in the United States is seeking a highly skilled Cloud Data Engineer to design, build, and optimize scalable data lakehouse and data warehousing solutions using Azure Databricks, Azure Synapse, and SQL. You will develop high-volume ETL/ELT pipelines and reusable Databricks notebooks, employing Python, PySpark, Spark SQL, and Scala while adhering to CI/CD practices.

The role supports AI/ML initiatives, feature engineering, and GenAI integrations, with strong emphasis on

Qualifications

• 7+ years of experience in cloud data engineering, ETL/ELT development, data warehousing, and enterprise analytics solutions.
• Strong hands-on experience with Azure Databricks, Azure Synapse Analytics, Azure Data Factory, and modern cloud-based data lakehouse architectures.
• Proven experience implementing the Medallion Architecture (Bronze, Silver, and Gold layers) to support scalable, governed, and high-quality data pipelines for enterprise analytics and reporting.
• Advanced proficiency in Python, PySpark, Spark SQL, SQL, T-SQL, and Scala, with the ability to develop, optimize, and support highly scalable Databricks notebooks and distributed data processing workloads.
• Proven experience designing, developing, and tuning complex queries and large-scale data processing solutions across high-volume datasets.
• Experience implementing CI/CD pipelines, Git-based source control, Azure DevOps, and automated deployment frameworks.

Responsibilities

• Design, develop, and maintain scalable cloud-based data engineering solutions, including ETL/ELT pipelines, data integration processes, reports, and analytics workflows using technologies such as Azure Databricks, Azure Synapse Analytics, Azure Data Factory, and SQL.
• Demonstrate expert knowledge of modern cloud architecture, data lakehouse design, and data warehousing best practices, evaluating technical design alternatives and their business and operational impacts.
• Develop, optimize, and troubleshoot complex SQL, T-SQL, Spark SQL, and PySpark workloads, with a strong focus on performance tuning, scalability, reliability, and cost efficiency.
• Design, code, test, tune, deploy, and document new and existing data solutions, including Databricks notebooks developed in Python, PySpark, Spark SQL, and Scala, as well as cloud-native applications, APIs, and data pipelines utilizing Git, CI/CD, and Azure DevOps best practices.
• Manage the complete development lifecycle for complex, high-impact data initiatives, including requirements gathering, solution design, development, testing, deployment, production support, and continuous improvement.
• Support and enable AI, machine learning, and Generative AI initiatives by developing curated datasets, feature engineering pipelines, and scalable data platforms that power advanced analytics and intelligent business solutions.
• Stay current with emerging trends and technologies in cloud computing, data engineering, artificial intelligence, Databricks, Azure services, automation, and analytics, recommending innovative solutions that drive business value and operational excellence.
• Collaborate effectively with business stakeholders, architects, developers, and leadership teams, leveraging exceptional verbal and written communication skills to translate business requirements into scalable technical solutions and communicate complex concepts to both technical and non-technical audiences.
• Perform other functionally related duties and responsibilities as assigned.

Skills

Python

PySpark

SQL

T-SQL

Scala

Databricks

Azure Databricks

Azure Synapse

ETL/ELT

CI/CD

Azure DevOps

Git

Data lakehouse

Data warehousing

Education

Bachelor's degree in CS/IS/Engineering

Tools

Azure Data Factory

CI/CD tooling

DevOps

Job description

AmTrust Financial Services in the United States is seeking a highly skilled Cloud Data Engineer to design, build, and optimize scalable data lakehouse and data warehousing solutions using Azure Databricks, Azure Synapse, and SQL. You will develop high-volume ETL/ELT pipelines and reusable Databricks notebooks, employing Python, PySpark, Spark SQL, and Scala while adhering to CI/CD practices.

The role supports AI/ML initiatives, feature engineering, and GenAI integrations, with strong emphasis on