Azure Data Engineer

Cloud Analytics Technologies, LLC; Austin (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Cloud Analytics Technologies, LLC in Austin, TX is seeking a Senior Data Engineer to design and implement scalable ETL/ELT pipelines on Azure. You will build data lakehouse solutions using ADLS Gen2, Synapse, Databricks, and Python-based transformations.

You will collaborate with data architects and stakeholders to ensure data quality, security, and governance, optimize Spark jobs and SQL queries, and implement CI/CD deployments with Azure DevOps.

Qualifications

• Strong experience with Microsoft Azure cloud data services and data engineering.
• Hands-on experience with Azure Data Factory (ADF) and Databricks.
• Proficient in Python and PySpark for data processing and scripting.
• Strong SQL skills and query optimization, with data warehousing knowledge.
• Experience with Azure Synapse Analytics and ADLS Gen2.
• Familiarity with data governance, security, and quality practices.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory.
• Build and optimize data processing solutions with Azure Databricks and PySpark.
• Develop data transformation workflows using Python and SQL.
• Implement data ingestion from multiple sources including databases, APIs, files, and cloud platforms.
• Build and maintain data lake and data warehouse solutions using ADLS Gen2 and Azure Synapse.
• Collaborate with data architects, analysts, and stakeholders to ensure data quality and compliance.
• Troubleshoot production data pipelines and ensure operational stability.

Skills

Azure data services

Azure Data Factory

Azure Databricks

Python

PySpark

SQL / query optimization

Azure Synapse Analytics

ADLS Gen2

Data warehousing

ETL/ELT pipelines

Azure DevOps

Data governance & security

Delta Lake

Tools

Git

Azure DevOps

Delta Lake

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

• Strong experience in Microsoft Azure cloud data services.
• Hands-on experience with Azure Data Factory (ADF).
• Strong experience with Azure Databricks.
• Good programming experience in Python and PySpark.
• Strong SQL skills and query optimization.
• Experience with Azure Synapse Analytics.
• Knowledge of Azure Data Lake Storage Gen2 (ADLS).
• Experience with data warehousing and data modeling concepts.
• Hands-on experience with data integration and transformation.
• Experience with Azure DevOps, Git, and CI/CD pipelines.
• Knowledge of data security, governance, and data quality.
• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory.
• Build and optimize data processing solutions using Azure Databricks and PySpark.
• Develop data transformation workflows using Python and SQL.
• Implement data ingestion from multiple sources, including databases, APIs, files, and cloud platforms.
• Build and maintain data lake and data warehouse solutions using ADLS Gen2 and Azure Synapse.
• Optimize Spark jobs, SQL queries, and data processing performance.
• Implement data validation, data quality checks, and error-handling mechanisms.
• Work with Delta Lake for reliable and scalable data storage.
• Implement CI/CD pipelines and deployment automation using Azure DevOps.
• Ensure data security, governance, and compliance standards.
• Collaborate with data architects, analysts, and business stakeholders.
• Troubleshoot production data pipeline issues and ensure operational stability.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Some of our Business Ventures - Knowledge Analytics

Do restrictive necessities, your capability for advanced analytics or your business model need custom-made applications and infrastructure configurations? opt for our hosted managed services to urge comprehensive analytics solutions supported industry-leading Cloud Analytics Technologies, LLC code - delivered by the Cloud Analytics Technologies, LLC consultants UN agency understand these solutions best. Through a mix of with-it technology and dedicated service and support, Cloud Analytics Technologies, LLC hosted managed services will facilitate you: