Senior Azure Data Factory Architect – Cloud & Data

TechDigital Group; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

An established industry player is seeking a Senior ADF Architect with over 15 years of experience in data engineering and cloud solutions. This role demands expertise in Azure Data Factory, Databricks, and Azure Cloud technologies. You will be responsible for designing and implementing robust data-oriented solutions, integrating on-premise and cloud environments, and applying your cloud engineering skills to solve complex problems. Join a collaborative team where your contributions will drive innovation and excellence in data management. If you are passionate about cloud solutions and have a proven track record, this opportunity is perfect for you.

Qualifications

• 15+ years of experience in data engineering and cloud solutions.
• Expertise in Azure Data Factory and related technologies.

Responsibilities

• Design and implement solutions using Azure Data Factory and Databricks.
• Integrate on-premise and cloud environments effectively.

Skills

Azure Data Factory

Databricks

Azure Cloud

Data transformations

Integration patterns (API, Messaging)

Cloud and data engineering skills

Communication skills

Tools

Azure Data Lake Storage

SQL Database

Azure Service Bus

Azure Logic Apps

Azure Functions

Job description

Job Description

• Administer, configure, maintain and upgrade GCP Composer (Apache Airflow) environment ( Mandatory )
• Design and implement infrastructure as code using Terraform for provisioning and managing GCP resources. (Mandatory)
• Able to create Terraform modules and publish same in TFE (Terraform Enterprise )
• DAG deployment, Troubleshooting, performance tuning, and operational support.
• Support and optimize Dataflow and Dataproc workloads, including batch and streaming data processing pipelines.
• Maintain GCP infrastructure including services like:
  • GCP Composer (Apache Airflow)
  • Google Cloud Storage (GCS)
  • BigQuery (BQ)
  • Cloud Dataflow /Apache Beam
  • Cloud Dataproc
  • Cloud Functions --- (Mandatory)
  • Cloud Run
  • Pub/Sub
  • Cloud SQL
  • Secret Manager
  • Logging & Monitoring
• Monitor GCP usage and quotas, and proactively prevent quota-related issues.
• Set up and manage alerting by Creating Alert policies in GCP Monitoring . (Mandatory)
• Manage CI/CD pipelines using Jenkins and/or using other CI/CD tools.(Mandatory)
• Able to request Identity and Access Management (IAM) permissions, service accounts, and security policies following least-privilege principles.
• Ensure security, compliance, and cost-efficiency across the cloud environment.
• Monitor cloud infrastructure health, performance, and resource utilization using Cloud Monitoring, Cloud Logging, Error Reporting, and Cloud Trace.
• Develop automation scripts and operational tools using Python and/or Java to improve platform efficiency and reduce manual effort.
• Create and maintain operational documentation, runbooks, architecture diagrams, standard operating procedures (SOPs), and platform best practices.
• Excellent problem-solving and troubleshooting skills.

Nice to Have Skills:

• Experience with CDC platforms such as Striim
• Experience with ServiceNow integrations and operational workflows
• Knowledge of FinOps, cloud cost optimization, and governance frameworks
• Networking components understanding including:
  • VPCs and Shared VPCs
  • VPC Connectors
  • Firewall Rules
  • CIDR Ranges

Exposure to healthcare domain