Senior Azure Data Engineer

kr3 information systems inc; Atlanta (GA); Remote; USD 140.000 - 170.000; Vollzeit

Zusammenfassung

kr3 information systems inc is seeking a hands-on Senior Azure Data Engineer to design and run production data pipelines on Azure from Atlanta. You will build lakehouse-style Bronze-Silver-Gold pipelines using ADF and Databricks, and validate data with Python, PySpark, Pandas, and Great Expectations.

Responsibilities include ingesting diverse file formats, triggering Databricks jobs, managing failures, and tuning pipelines in production while ensuring data quality for downstream users.

Qualifikationen

• 7+ years in data engineering
• Hands-on experience with Azure Data Factory and Azure Databricks
• Strong Python, PySpark, and SQL skills
• Experience building data validation frameworks with Great Expectations and Pandas
• Solid cloud data engineering background; Azure preferred; AWS or Google Cloud considered

Aufgaben

• Analyze incoming data files (CSV, JSON, Parquet) and their schemas to plan ingestion and transformation.
• Design and build lakehouse-style Medallion pipelines (Bronze, Silver, Gold) using Azure Data Factory for movement and Azure Databricks for processing.
• Build and maintain multi-step pipelines in Azure Data Factory.
• Use ADF to trigger/manage Azure Databricks Jobs, pass parameters, track status, handle failures and retries.
• Write data transformation and validation logic in Python and PySpark; set up automated data quality checks with Great Expectations and Pandas.
• Troubleshoot and tune pipelines in production.

Kenntnisse

Azure Data Factory

Azure Databricks

Python

PySpark

Pandas

Great Expectations

SQL

Data validation

Tools

Azure Data Factory

Azure Databricks

Great Expectations

Jobbeschreibung

Senior Azure Data Engineer

Location: Atlanta, GA (Open for Remote)

Type: [Full-time / Contract] | Start: Immediate

We're looking for a hands-on Senior Azure Data Engineer to build and run production data pipelines on Azure. This role focuses on building complex data pipelines using Azure Data Factory and Azure Databricks, along with data validation using Python, PySpark, Pandas, and Great Expectations.

What You'll Do:

• Analyze incoming data files (CSV, JSON, Parquet) and their schemas to plan ingestion and transformation.
• Design and build lakehouse-style Medallion (Bronze, Silver, Gold) data pipelines, using Azure Data Factory for data movement and Azure Databricks for complex data processing.
• Build and maintain complex, multi-step pipelines in Azure Data Factory.
• Use ADF to trigger and manage Azure Databricks Jobs, including passing parameters, tracking job status, and handling failures and retries.
• Write data transformation and validation logic in Python and PySpark.Set up automated data quality checks using Great Expectations and Pandas, and flag bad data before it reaches downstream users.
• Troubleshoot and tune pipelines in production.

Core Skills Required-

1. 7+ years in data engineering, with recent senior-level work.
2. Strong hands-on experience with Azure Data Factory and its integration with Azure Databricks Jobs.Strong Python, PySpark, and SQL skills.
3. Hands-on experience building data validation frameworks with
4. Great Expectations and Pandas.
5. Solid cloud data engineering background. Azure preferred; AWS or Google Cloud Platform considered.

Big Plus-

1. Local to Atlanta, GA / Open for remote too
2. Experience on State or Federal government projects.
3. Azure Data Engineer (DP-203) or Databricks certification.