Data Engineer

SoftEdge; India; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

SoftEdge in India is seeking a Client Data Engineer to design, develop, validate, optimize, and own data solutions using AWS and Databricks. You will partner with internal teams, governance, BI/analytics, and implementation partners to deliver scalable data pipelines and reporting-ready data foundations.

The ideal candidate has 5+ years in data engineering, ETL/ELT, and data integration, with hands-on Databricks and AWS experience, plus strong SQL, Python, PySpark, and Spark skills.

Qualifications

• 5+ years of experience in data engineering, ETL/ELT, and data integration.
• Strong hands-on Databricks experience.
• Experience with AWS data services and cloud platforms.
• Strong SQL and Python skills.
• Experience with PySpark / Apache Spark.

Responsibilities

• Design, develop, and support AWS and Databricks data pipelines.
• Develop and maintain Bronze, Silver, and Gold data layers using Databricks and Delta Lake.
• Review and validate data solutions with implementation partners.
• Perform data profiling, reconciliation, testing, and troubleshooting.
• Collaborate with BI and analytics teams on reporting needs.
• Support data quality, metadata, lineage, and Unity Catalog activities.
• Optimize pipelines for performance, scalability, and cost efficiency.
• Participate in CI/CD, deployment, and production support.

Skills

Data engineering

ETL/ELT

SQL

Python

PySpark

Data integration

Troubleshooting

Tools

Databricks

AWS

Tableau

Power BI

Delta Lake

Unity Catalog

Spark

Job description

The Client Data Engineer will be responsible for the design, development, validation, optimization, and ongoing ownership of data solutions primarily using AWS and Databricks. The role will work closely with internal teams, business SMEs, architects, governance teams, Power BI/analytics teams, and implementation partners.

The ideal candidate should have 5+ years of experience in data engineering, ETL/ELT, and data integration, with strong hands‑on expertise in Databricks, AWS, SQL, Python, PySpark, and Spark.

Key Responsibilities

• Design, develop, and support AWS and Databricks data pipelines using SQL, Python, PySpark, and Spark.
• Develop and maintain Bronze, Silver, and Gold data layers using Databricks and Delta Lake.
• Work with source-system data, business rules, mappings, and integration requirements.
• Review and validate data solutions delivered by implementation partners.
• Perform data profiling, reconciliation, testing, and troubleshooting.
• Support data quality, metadata, lineage, and Unity Catalog activities.
• Optimize Databricks pipelines and queries for performance, scalability, and cost efficiency.
• Participate in code reviews, CI/CD, deployment, and production support.
• Maintain technical documentation and support knowledge transfer to internal teams.
• Collaborate closely with BI and analytics teams on downstream reporting and data requirements.
• Support integration with Tableau and Power BI for reporting and analytics use cases.

Required Skills

• 5+ years of experience in data engineering, ETL/ELT, or data integration.
• Strong hands‑on experience with Databricks.
• Strong experience with AWS data services and cloud‑based data platforms.
• Strong SQL and Python skills.
• Strong experience with PySpark / Apache Spark.
• Experience designing and implementing data pipelines and data transformation frameworks.
• Strong understanding of data integration, transformation, reconciliation, and data quality.
• Strong analytical, troubleshooting, and communication skills.

Preferred Skills

• Databricks, Delta Lake, and Unity Catalog.
• AWS Databricks / Databricks on AWS experience.
• Medallion/Lakehouse architecture.
• Experience with Tableau reporting and Power BI.
• Git, CI/CD, and DevOps practices.
• Experience with data governance, metadata, lineage, and data quality frameworks.