Azure Data Scientist & Lakehouse Engineer (Hybrid)

Brilliant Infotech Inc.; Indianapolis (IN); Hybrid; USD 83,000 - 124,000; Full time

Benefits offered by this job

Hybrid work model

12-month contract ROP

Job summary

Brilliant Infotech Inc. is seeking a Data Scientist/Data Engineer with hands-on Azure Databricks, PySpark, Spark SQL and Delta Lake experience to develop and optimize enterprise data pipelines and lakehouse solutions.

The role focuses on data integration, governance, security, and analytics for sensitive agency data in a hybrid Indianapolis setting. Applicants should have strong Azure Databricks and Synapse skills, proficiency in SQL and dimensional modeling, and experience with CI/CD and data

Qualifications

• Hands-on experience with Azure Databricks, PySpark, Spark SQL and Delta Lake.
• Experience with medallion lakehouse architecture, batch/streaming pipelines, and incremental ingestion.
• Strong Azure Synapse Analytics, SQL, dimensional modeling and warehouse optimization.
• Experience with Azure Data Factory, Databricks Workflows and Unity Catalog.
• Knowledge of ADLS Gen2, Entra ID, Key Vault, governance and security controls.
• Experience with Git, CI/CD, Azure DevOps/GitHub, monitoring and troubleshooting.
• Strong communication and technical documentation skills.

Responsibilities

• Develop and optimize enterprise data pipelines and lakehouse solutions.
• Design and tune data warehouse workloads for performance and scalability.
• Support data governance, security and analytics for sensitive agency data.
• Collaborate with stakeholders and document architectures and workflows.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Lakehouse architecture

Batch/Streaming pipelines

Azure Synapse Analytics

SQL

Dimensional modeling

Data warehousing optimization

Azure Data Factory

Databricks Workflows

Unity Catalog

ADLS Gen2

Microsoft Entra ID

Key Vault

Data governance

Security controls

Git

CI/CD

Azure DevOps

Monitoring

Troubleshooting

Documentation

Communication

Tools

Power BI

Databricks Certification

Job description

Brilliant Infotech Inc. is seeking a Data Scientist/Data Engineer with hands-on Azure Databricks, PySpark, Spark SQL and Delta Lake experience to develop and optimize enterprise data pipelines and lakehouse solutions.

The role focuses on data integration, governance, security, and analytics for sensitive agency data in a hybrid Indianapolis setting. Applicants should have strong Azure Databricks and Synapse skills, proficiency in SQL and dimensional modeling, and experience with CI/CD and data