Senior DataOps Engineer

Amtex Enterprises Inc; Charlotte (NC); Hybrid; USD 110,208 - 117,096; Full time

Job summary

Amtex Enterprises Inc. seeks a Senior DataOps Engineer to design, build, and optimize enterprise-scale data platforms using Databricks and AWS. You will drive platform automation, IaC, and DataOps practices while supporting analytics and AI/ML initiatives.

The role emphasizes administering Databricks environments, CI/CD pipelines, and governance across Bronze-Silver-Gold lakehouse architectures with emphasis on performance and reliability.

Qualifications

• 8+ years of data engineering or data platform experience in enterprise settings.
• 5+ years hands-on Databricks administration in production environments.
• Strong Python, PySpark, Spark SQL, and SQL proficiency.
• Deep knowledge of Delta Lake, Unity Catalog, and cluster tuning.
• Terraform and IaC for cloud infrastructure provisioning.
• CI/CD and DevOps/DataOps practices in modern data platforms.
• Extensive AWS services experience: AWS Glue, Kinesis, Firehose, S3, IAM.
• Experience with Lakehouse/Medallion architectures and governance.

Responsibilities

• Design, build, and support enterprise-scale ETL/ELT pipelines in Databricks.
• Develop batch and streaming data pipelines using PySpark and Delta Lake.
• Configure Databricks clusters for performance, cost, and reliability.
• Administer Unity Catalog, governance, access control, lineage, and security.
• Implement Delta Lake best practices, partitioning, and schema evolution.
• Build and maintain IaC with Terraform for Databricks and AWS infra.
• Develop CI/CD pipelines and automate deployments for data workloads.
• Collaborate with analytics, AI/ML, and engineering teams on data solutions.
• Monitor, troubleshoot, and optimize Databricks jobs and workflows.
• Provide technical leadership and mentoring in DataOps best practices.

Skills

Databricks

Python

PySpark

Spark SQL

SQL

Delta Lake

Unity Catalog

Terraform

IaC

CI/CD

AWS

Kinesis

Firehose

S3

IAM

DataOps

Tools

None

Job description

Job Title: Senior DataOps Engineer

Duration: 12+ Months

Rate: $80-85/hr on Vendor W2- MAX

Location: Remote or Hybrid (Charlotte, NC)

Department: Integration, Data & AI Engineering

Position Overview

Seeking a Senior DataOps Engineer to design, build, automate, and support enterprise-scale data platforms leveraging Databricks and AWS. This role is responsible for developing scalable, high-performing, and governed data pipelines while driving platform automation, Infrastructure as Code (IaC), and DataOps best practices.

The ideal candidate will have extensive experience administering Databricks environments, optimizing platform performance, implementing CI/CD pipelines, and supporting enterprise analytics and AI/ML initiatives.

Key Responsibilities

• Databricks Platform Engineering & DataOps
• Design, build, optimize, and support enterprise-scale ETL/ELT pipelines within Databricks.
• Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
• Configure and optimize Databricks clusters for performance, scalability, reliability, and cost efficiency.
• Implement Delta Lake best practices, including partitioning, schema evolution, compaction, optimization, and performance tuning.
• Administer Unity Catalog, including governance, access controls, auditing, lineage, and security.
• Design and support Medallion (Bronze, Silver, Gold) Lakehouse architectures.
• Monitor, troubleshoot, and optimize Databricks jobs, workflows, and platform services.
• Support enterprise AI/ML and analytics workloads running within Databricks.
• Cloud Data Engineering
• Build and maintain scalable data ingestion and transformation pipelines using Python, PySpark, SQL, AWS Glue, and other AWS services.
• Integrate structured, semi-structured, unstructured, and streaming data from enterprise systems.
• Develop real-time data processing solutions using AWS Kinesis and Firehose.
• Partner with architecture, analytics, AI/ML, and engineering teams to deliver enterprise data solutions.
• Infrastructure Automation & DevOps
• Implement Infrastructure as Code (IaC) using Terraform to provision and manage Databricks environments and AWS infrastructure.
• Automate deployments, environment provisioning, configuration management, and operational workflows.
• Design and maintain CI/CD pipelines supporting Databricks deployments and infrastructure automation.
• Manage version control repositories and DataOps best practices.
• Drive platform standardization and deployment consistency across development, test, and production environments.
• Governance & Operational Excellence
• Ensure compliance with enterprise security, governance, privacy, and regulatory standards.
• Implement data quality controls, lineage tracking, auditing, and operational monitoring.
• Develop operational standards, monitoring frameworks, and support procedures.
• Provide technical leadership and mentor engineers on DataOps and Databricks best practices.

Required Qualifications

• 8+ years of experience in Data Engineering, Platform Engineering, or DataOps.
• 5+ years of hands‑on experience with Databricks in enterprise environments.
• Strong experience with Python, PySpark, Spark SQL, and SQL.
• Deep expertise with Delta Lake, Databricks Workflows, Unity Catalog, cluster administration, and performance optimization.
• Experience designing and supporting Lakehouse/Medallion architectures.
• Proven experience with Terraform and Infrastructure as Code (IaC).
• Strong knowledge of CI/CD pipelines and DevOps/DataOps methodologies.
• Experience with AWS services including AWS Glue, Kinesis, Firehose, S3, and IAM.
• Strong understanding of data governance, security, observability, and monitoring.
• Excellent communication, leadership, problem‑solving, and collaboration skills.

Deliverables

• Production‑ready Databricks ETL/ELT pipelines and workflows.
• Optimized and governed Databricks platform environments.
• Terraform modules and Infrastructure as Code automation.
• Monitoring and observability dashboards for Databricks workloads.
• Enterprise data models, lineage documentation, and operational runbooks.
• CI/CD pipelines and deployment automation.
• Weekly status updates and participation in Agile ceremonies.