Data Engineer

Tech SMCSquared GCC; Hyderabad; Hybrid; INR 1,800,000 - 2,400,000; Full time

Job summary

Tech SMCSquared GCC is seeking a data engineering professional to design, build, and maintain billing-oriented data pipelines in Azure and Databricks. You will transform source data into trusted, curated datasets and implement rigorous data quality checks to ensure accuracy and compliance.

The role requires strong Python/PySpark skills, extensive SQL, and experience with batch orchestration tools like ActiveBatch.

Qualifications

• 3–6 years of hands-on data engineering experience.
• Bachelor’s degree in a technical field or equivalent.
• Experience with secure handling of financial data.
• Hybrid/global engineering environment experience.
• Billing/payments financial domain experience preferred.

Responsibilities

• Design, build, test, deploy, and maintain ETL/ELT pipelines in Azure/Databricks.
• Transform source data into trusted, billing-ready datasets.
• Implement data quality validations and reconciliations.
• Develop and support ActiveBatch schedules, dependencies, and recovery procedures.
• Build integrations with third-party providers and internal systems.
• Monitor pipelines and respond to failures with root-cause analysis.
• Ensure timely delivery within billing windows.

Skills

Databricks

Python

PySpark

SQL

Azure

ActiveBatch

Data quality checks

Education

Bachelor's degree in CS/IS/Engineering

Tools

ActiveBatch

Snowflake (bonus)

Git workflows

CI/CD

Job description

Key Responsibilities

• Billing Data Engineering: Design, build, test, deploy, and maintain reliable ETL/ELT pipelines and reusable data components in Azure and Databricks using Python, PySpark, and SQL.
• Data Hub & Curation: Transform source data into trusted, billing-ready curated datasets. Maintain consistent business rules, source-to-target traceability, and accurate delivery across data hub and curation layers.
• Data Quality & Reconciliation: Implement rigorous automated validations and reconciliations for completeness, accuracy, consistency, duplicates, referential integrity, and expected volumes. Treat data quality defects as production-impacting issues and resolve them with urgency.
• Orchestration & Scheduling: Develop and support ActiveBatch schedules, dependencies, execution chains, reruns, and recovery procedures. Ensure pipelines complete within narrow billing windows and downstream delivery commitments.
• Third-Party & Internal Integrations: Build and operate integrations with third-party providers and internal systems using APIs, databases, secure file transfers, and other appropriate interfaces. Design for retries, recoverability, idempotency, and clear failure handling.
• Production Reliability & Incident Response: Monitor pipeline execution and data delivery; investigate failures, data anomalies, and missed dependencies immediately. Apply safe in-window fixes or workarounds when necessary, communicate impact and status, and follow through with root-cause and permanent corrective actions.
• Performance & Timeliness: Optimize jobs and workflows to meet strict processing deadlines. Identify bottlenecks early, manage competing priorities, and upscale risks before they threaten billing timelines.
• Problem Solving & Alternatives: Break down ambiguous technical and data issues methodically, validate assumptions, and propose alternative implementation paths when constraints or dependencies prevent the preferred solution.
• Business & Technical Collaboration: Partner with Billing, Operations, Product, engineering, data, and other stakeholders to clarify requirements, understand business rules, communicate tradeoffs, and translate needs into dependable technical solutions.
• Security & Sensitive Data Handling: Follow secure engineering and data-handling practices for sensitive financial and billing data, including least-privilege access, appropriate controls, auditability, and protection of confidential information.
• Engineering Practices: Use Git-based workflows, code reviews, automated testing, deployment standards, documentation, and maintainable design patterns. Build solutions that can be supported under production time pressure without sacrificing quality.
• Continuous Improvement: Improve pipeline reliability, operational runbooks, test coverage, observability, recovery procedures, and team knowledge sharing based on production learnings and recurring failure patterns.

Knowledge, Skills, Abilities

• Strong hands-on experience with Databricks for production data engineering workloads.
• Strong Python and PySpark skills for data transformation, automation, validation, troubleshooting, and operational support.
• Strong SQL skills with the ability to analyze, transform, reconcile, and validate complex datasets.
• Practical experience designing, building, and supporting ETL/ELT pipelines in Microsoft Azure.
• Experience with job orchestration, scheduling, dependencies, and recovery patterns; ActiveBatch experience is preferred, or equivalent experience with enterprise batch/orchestration tools.
• Experience integrating data from third-party providers and internal systems through APIs, databases, and file-transfer patterns.
• Demonstrated ability to implement high-confidence data quality checks, source-to-target validation, reconciliation controls, and production-ready error handling.
• Strong understanding of curated data layers, data modeling, dependency management, schema evolution, and reliable downstream delivery.
• Ability to troubleshoot production data issues quickly, isolate root causes, make sound decisions under time pressure, and implement safe corrective actions.
• Strong time management and prioritization skills, including the ability to deliver within narrow, non-negotiable processing windows and manage multiple urgent dependencies.
• Excellent written and verbal communication skills with the ability to provide concise status, explain technical issues to non-technical partners, and raise risks early.
• Strong problem-solving mindset with a willingness to challenge assumptions and propose practical alternative solutions when needed.
• Experience with version control, collaborative Git workflows, automated testing, and CI/CD practices.
• Understanding of secure handling of sensitive or regulated financial data, access controls, privacy, and auditability.
• Exposure to Snowflake is a plus for the team's future-state evolution, but Snowflake experience is not required for the current environment.
• Experience in billing, payments, financial technology, wealth management, asset management, or another high-control financial domain is strongly preferred.

Qualifications & Experience

• 3-6 years of hands‑on experience in data engineering, data integration, software engineering, or a closely related technical role, including production support responsibilities.
• Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, Statistics, or a related technical field, or equivalent practical experience.
• Demonstrated experience supporting business‑critical data pipelines with strict quality and delivery expectations.
• Experience working with sensitive or regulated financial data and following secure data‑handling practices is required.
• Ability to work effectively in a hybrid, globally distributed engineering organization and collaborate across technical and business teams.