Azure Databricks Engineer

Artifintech; Houston (TX); On-site; USD 130,000 - 170,000; Full time

Job summary

ARTIFICIAL INTELLIGENCE TECHNOLOGIES LLC is seeking a Specialist in Houston, TX with 7 years of experience designing and developing scalable data solutions using Azure Databricks. You will design, develop, and optimize data pipelines to support enterprise-level analytics and data processing.

Collaborate with cross-functional teams to align data solutions with business objectives and ensure high data quality and reliability.

Qualifications

• 7+ years of experience designing and developing scalable data solutions with Azure Databricks.
• Proficient in PySpark and Python for ETL, ingestion, transformation and analytics.
• Experience designing scalable data pipelines and integrating Azure data services.
• Ability to collaborate with cross-functional teams and align data solutions with business goals.

Responsibilities

• Lead design, development and optimization of complex data pipelines on Azure Databricks.
• Write clean PySpark and Python code for data ingestion, transformation and analytics.
• Troubleshoot pipelines for minimal downtime and high availability.
• Document data processes, architectures and workflows for knowledge sharing.
• Mentor junior engineers and participate in design reviews and agile ceremonies.

Skills

Azure Databricks

PySpark

Python

Data pipelines

Tools

Azure Databricks

ETL tools

Job description

Seeking a Specialist with 7 years of experience in designing and developing scalabledatasolutions using AzureDatabricks

Design develop and optimize scalabledatapipelines leveraging AzureDatabricksto support enterprise leveldataprocessing andanalyticsImplement and maintain ETL workflows ensuring high performancedataquality and reliability Integrate AzureDatabrickswith various Azuredataservices to build robust and scalabledataplatforms Monitor and optimizedataworkflows for scalability cost effectiveness and operational efficiency Collaborate with cross functional teams to aligndatasolutions with business objectives and technical requirements Stay updated with the latest advancements in AzureDatabricksand recommend best practices for continuous improvement

Roles and Responsibilities

Lead the design development and enhancement of complexdatapipelines using AzureDatabricksWrite clean maintainable and efficient PySpark and Python code to supportdataingestion transformation andanalyticsTroubleshoot and resolve issues related todatapipelines and workflows to ensure minimal downtime and high availability Collaborate withdataarchitectsanalystsand stakeholders to gather requirements and deliver scalabledatasolutions Mentor and guide junior team members on AzureDatabricksand best practices indataengineering Documentdataprocesses architectures and workflows to ensure knowledge sharing and compliance Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and highperforming team environment

ARTIFICIAL INTELLIGENCE TECHNOLOGIES LLC is an equal opportunity employer inclusive of female, minority, disability and veterans, (M/F/D/V). Hiring, promotion, transfer, compensation, benefits, discipline, termination and all other employment decisions are made without regard to race, color, religion, sex, sexual orientation, gender identity, age, disability, national origin, citizenship/immigration status, veteran status or any other protected status. ARTIFICIAL INTELLIGENCE TECHNOLOGIES LLC will not make any posting or employment decision that does not comply with applicable laws relating to labor and employment, equal opportunity, employment eligibility requirements or related matters. Nor will ARTIFICIAL INTELLIGENCE TECHNOLOGIES LLC require in a posting or otherwise U.S. citizenship or lawful permanent residency in the U.S. as a condition of employment except as necessary to comply with law, regulation, executive order, or federal, state, or local government contract