Databricks Architect

Inizio Partners Corp; New York (NY); On-site; USD 140,000 - 220,000; Full time

Job summary

Inizio Partners Corp is seeking a Senior Data Engineer to design and optimize ETL pipelines on cloud Databricks, with a focus on insurance data and real-time integration via Spark. You will lead data engineering projects, collaborate with global teams, and implement automated testing and CI/CD practices.

The role requires strong SQL/Python skills, experience with Databricks, Delta Lake, and Spark, plus familiarity with Airflow, Splunk, Kubernetes, and Azure DevOps.

Qualifications

• 8+ years creating and implementing ETL pipelines using Databricks on cloud.
• Some experience in insurance domain data is must.
• Programming Languages – SQL, Python.
• Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).
• Good to have - Airflow, Splunk, Kubernetes, Power BI, Git, Azure DevOps.
• B.S. Degree in a data-centric field (Mathematics, Economics, CS, Engineering or related).
• Excellent communication & leadership skills, with the ability to lead and motivate team members.

Responsibilities

• Develop and optimize ETL pipelines on cloud Databricks.
• Implement automated testing and Airflow scheduling.
• Leverage Azure DevOps for CI/CD and Terraform for infrastructure as code.
• Monitor operations and optimize compute/storage costs with Splunk.
• Lead global delivery teams using Scrum/Agile ceremonies and manage backlog.
• Integrate data across different systems and platforms.
• Maintain clear client communications and project documentation.

Skills

ETL pipelines

Databricks

Cloud platforms

SQL

Python

Airflow

Splunk

Kubernetes

Power BI

Git

Azure DevOps

Communication

Leadership

Agile/Scrum

Education

B.S. Degree

Data-centric field

Tools

Databricks

Delta Lake

Spark

Airflow

Splunk

Kubernetes

Power BI

Git

Azure DevOps

Job description

• Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)
• Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring
• Continuously improve systems through performance enhancements and cost reductions in compute and storage
• Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs
• Lead Data Engineering Projects to manage and implement data-driven communication systems
• Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations
• Integrate data across different systems and platforms
• Strong verbal and written communication skills to manage client discussions

Candidate Profile:

• 8+ years experience in developing and implementing ETL pipelines from various data sources using Databricks on cloud
• Some experience in insurance domain/ data is must
• Programming Languages – SQL, Python
• Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).
  • Good to have - Airflow, Splunk, Kubernetes, Power BI, Git, Azure DevOps
• B.S. Degree in a data-centric field (Mathematics, Economics, Computer Science, Engineering or other science field), Information Systems, Information Processing or engineering.
• Excellent communication & leadership skills, with the ability to lead and motivate team members
• Ability to work independently with some level of ambiguity and juggle multiple demands