Azure Databricks Data Engineer

Sutherland; United States; Remote; USD 80,000 - 120,000; Full time

Job summary

An innovative company is seeking a skilled Data Engineer to build scalable data pipelines and optimize ETL processes in Azure cloud environments. This role involves working with cutting-edge technologies like Databricks and Azure Data Factory to manage and transform data effectively. You'll collaborate with business analysts and data scientists to ensure high data quality and governance while implementing robust data models. If you thrive in a dynamic environment and have a passion for big data and distributed computing, this position offers a fantastic opportunity to make a significant impact.

Qualifications

• Strong experience with Azure Data Factory and Databricks for data processing.
• Proficiency in Python and SQL for ETL and data manipulation.

Responsibilities

• Develop and optimize ETL pipelines using ADF and Databricks.
• Design data models for structured and unstructured data.

Skills

Azure Data Factory (ADF)

Databricks

Python

SQL

Apache Spark

Data Governance

Performance Optimization

Tools

Azure Synapse

Delta Lake

CI/CD

Power BI

Tableau

Job description

Job Title: Data Engineer

Job Summary:

Are you passionate about building scalable data pipelines, optimizing ETL processes, and designing efficient data models? We are looking for a Databricks Data Engineer to join our team and play a key role in managing and transforming data in Azure cloud environments.

In this role, you will work with Azure Data Factory (ADF), Databricks, Python, and SQL to develop robust data ingestion and transformation workflows. You will also be responsible for integrating, optimizing performance, and ensuring data quality & governance. If you have strong experience in big data processing, distributed computing (Spark), and data modeling, we’d love to hear from you!

Key Responsibilities:

1. Develop & Optimize ETL Pipelines:
   • Build robust and scalable data pipelines using ADF, Databricks, and Python for data ingestion, transformation, and loading.
2. Data Modeling & Systematic Layer Modeling:
   • Design logical, physical, and systematic data models for structured and unstructured data.
3. Database Management:
   • Develop and optimize SQL queries, stored procedures, and indexing strategies to enhance performance.
4. Big Data Processing:
   • Work with Azure Databricks for distributed computing, Spark for large-scale processing, and Delta Lake for optimized storage.
5. Data Quality & Governance:
   • Implement data validation, lineage tracking, and security measures for high-quality, compliant data.
6. Collaboration:
   • Work closely with business analysts, data scientists, and DevOps teams to ensure data availability and usability.
7. Testing and Debugging:
   • Write unit tests and perform debugging to ensure the implementation is robust and error-free.
   • Conduct performance optimization and security audits.

Required Skills and Qualifications:

• Azure Cloud Expertise: Strong experience in Azure Data Factory (ADF), Databricks, and Azure Synapse.
• Programming: Proficiency in Python for data processing, automation, and scripting.
• SQL & Database Skills: Advanced knowledge of SQL, T-SQL, or PL/SQL for data manipulation.
• Data Modeling: Hands-on experience in dimensional modeling, systematic layer modeling, and entity-relationship modeling.
• Big Data Frameworks: Strong understanding of Apache Spark, Delta Lake, and distributed computing.
• Performance Optimization: Expertise in query optimization, indexing, and performance tuning.
• Data Governance & Security: Knowledge of RBAC, encryption, and data privacy standards.

Preferred Qualifications:

1. Experience with CI/CD for data pipelines using Azure DevOps.
2. Knowledge of Kafka/Event Hub for real-time data processing.
3. Experience with Power BI/Tableau for data visualization (not mandatory but a plus).