Azure Data Engineer

Ethereum Technologies; Austin (TX); Vor Ort; USD 90.000 - 130.000; Vollzeit

Zusammenfassung

Ethereum Technologies in Austin is seeking a skilled data engineer to design and optimize Azure-based ETL/ELT pipelines. You will work with Azure Data Factory, Databricks, PySpark, and SQL to transform data from multiple sources into scalable lakehouse solutions.

The role requires hands-on experience with ADLS Gen2, Delta Lake, and CI/CD practices, collaborating with data teams and stakeholders to ensure data quality and governance.

Qualifikationen

• Strong experience with Azure cloud data services and data pipelines.
• Hands-on with ADF and Databricks for ETL/ELT design.
• Proficient in Python, PySpark and SQL optimization.
• Experience with ADLS Gen2, Synapse Analytics, and data modeling.
• Familiar with CI/CD pipelines and data security/governance basics.
• Knowledge of Delta Lake and scalable data lake/warehouse design.

Aufgaben

• Design, develop, and maintain scalable ETL/ELT pipelines using ADF.
• Build and optimize data processing with Azure Databricks and PySpark.
• Develop data transformation workflows using Python and SQL.
• Ingest data from databases, APIs, files, and cloud platforms.
• Build data lake and data warehouse solutions with ADLS Gen2 and Synapse.
• Tune Spark jobs and SQL for performance; implement data quality checks.
• Collaborate with data architects and stakeholders; enable CI/CD deployment.
• Ensure data security, governance, and compliance across pipelines.

Kenntnisse

Azure cloud data services

Azure Data Factory (ADF)

Azure Databricks

Python

PySpark

SQL

Delta Lake

Azure Synapse Analytics

ADLS Gen2

CI/CD

Tools

Azure DevOps

Git

Delta Lake

Jobbeschreibung

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

• Strong experience in Microsoft Azure cloud data services.
• Hands-on experience with Azure Data Factory (ADF).
• Strong experience with Azure Databricks.
• Good programming experience in Python and PySpark.
• Strong SQL skills and query optimization.
• Experience with Azure Synapse Analytics.
• Knowledge of Azure Data Lake Storage Gen2 (ADLS).
• Experience with data warehousing and data modeling concepts.
• Hands-on experience with data integration and transformation.
• Experience with Azure DevOps, Git, and CI/CD pipelines.
• Knowledge of data security, governance, and data quality.
• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory.
• Build and optimize data processing solutions using Azure Databricks and PySpark.
• Develop data transformation workflows using Python and SQL.
• Implement data ingestion from multiple sources, including databases, APIs, files, and cloud platforms.
• Build and maintain data lake and data warehouse solutions using ADLS Gen2 and Azure Synapse.
• Optimize Spark jobs, SQL queries, and data processing performance.
• Implement data validation, data quality checks, and error-handling mechanisms.
• Work with Delta Lake for reliable and scalable data storage.
• Implement CI/CD pipelines and deployment automation using Azure DevOps.
• Ensure data security, governance, and compliance standards.
• Collaborate with data architects, analysts, and business stakeholders.
• Troubleshoot production data pipeline issues and ensure operational stability.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.