Custom Software Engineering Lead

Accenture in India; Maharashtra; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Accenture in India is seeking a Custom Software Engineering Lead to own the technical direction and architecture of complex data solutions. You will guide design and delivery, setting standards for performance and scalability while aligning with business goals.

You will mentor teams, drive architecture decisions, and collaborate across functions to deliver high-impact results. The role emphasizes deep expertise in Azure Databricks, Spark-based data platforms, and end-to-end data solutions.

Qualifications

• Bachelor's degree or higher in Computer Science/Engineering/Science.
• 8–10 years data engineering experience with 5+ years in Azure cloud native services.
• Experience designing large-scale data solutions in Azure Databricks for batch and real-time analytics.
• Strong knowledge of Lakehouse, Delta Lake, Data Mesh, and Data Warehouse concepts.
• Proficient in Python/PySpark and data engineering libraries.
• Experience with distributed data platforms (Spark, Kafka, Flink) at petabyte scale.
• Experience enabling pipelines to downstream BI/data science needs.
• Familiar with Docker, Airflow, and Azure Data Factory.

Responsibilities

• Act as SME, collaborate and manage the team to perform.
• Responsible for team decisions and architecture choices.
• Engage with multiple teams and contribute on key decisions.
• Provide solutions to problems for multiple teams.
• Drive continuous improvement of software development processes and productivity.
• Mentor and support junior team members to grow skills.

Skills

Azure Databricks expertise

Python/PySpark

Big data processing

Distributed data pipelines

Education

15 years full time education

Tools

Azure Databricks

Azure Data Factory

Airflow

Job description

Project Role :

Custom Software Engineering Lead

Project Role Description :

Own the technical direction and architecture of custom software solutions, leading teams through design and delivery. Set development standards and ensure code quality, scalability, and performance aligned to business objectives.

Must have skills :

Microsoft Azure Databricks

Good to have skills :

NA

Minimum 7.5 Year(s) Of Experience Is Required

Educational Qualification :

15 years full time education

Summary :

As a Custom Software Engineering Lead, a typical day involves overseeing the technical direction and architectural design of custom software solutions. This role requires guiding and supporting teams throughout the design and delivery phases, ensuring that development standards are maintained. The position focuses on aligning software performance and scalability with business goals, while fostering collaboration and innovation within the team to meet project objectives effectively.

Roles & Responsibilities :

• Expected to be an SME, collaborate and manage the team to perform.
• Responsible for team decisions.
• Engage with multiple teams and contribute on key decisions.
• Provide solutions to problems for their immediate team and across multiple teams.
• Drive continuous improvement initiatives to enhance software development processes and team productivity.
• Mentor and support junior team members to foster professional growth and skill development.

Experience & Skills :

• a bachelor s degree or higher in Computer Science/ Engineering/ Science
• data engineer with 8 to 10 years of experience, having 5+ experience in Azure cloud native services.
• Experience in implementing and Designing Big Data solutions in Azure Databricks for Batch and Real Time Analytics uses cases.
• Experience in core concepts like Lakehouse, Delta Lake, Data Mesh, Data Virtualization, Dimensional Modelling techniques, Data Aggregation techniques.
• Experience in programming skills in languages such as Python/Pyspark and frameworks and libraries based on them with a focus on data engineering.
• Extensive experience designing, building, and operating robust distributed data platforms (e.g., Spark, Kafka, Flink, HBase) and handling data at the petabyte scale.
• Experience in data engineering on Apache Spark and populate downstream batch data stores (such as data mart) for BI use cases & to generate downstream feeds (ie., flat & wide tables or compressed files) for Data Science use cases
• Experience in Real-time service integration to process business events off Kafka and persist in ADLS Gen 2 using Databricks.
• Experience in Near real-time stream processing to derive features for ML model inference.
• Experience in Hadoop based implementations, Python, Spark, SQL, Unix, and Hive
• Extensive experience and knowledge of a variety of data technologies and frameworks such as Delta Lake, Datahub, Apache Spark, Azure Databricks, Databricks Genie
• Strong familiarity with the concepts of RDBMS, Data Lake, Data Warehouse, Data Lakehouse, and Medallion Architecture.
• Knowledge of at least one workflow orchestration tools such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks
• Experience in Azure data bricks performance optimization techniques (Liquid clustering, vacuums, Z-ordering etc)
• Experience in Azure Services like Azure Data Factory, Azure Data Bricks (Scala/Python), ADLS Gen2, Azure SQL Database, SQL/T-SQL
• Experience working with the DevOps teams to drive operationally excellent infrastructure.
• Knowledge on Microsoft Fabric and its framework
• Basic understanding of data quality dimensions like Consistency, Completeness, Accuracy, Lineage etc.
• Understanding on different monitoring and logging solutions on Azure
• Good to have experience on Data Visualization Tools like Power BI and Tableau
• Good to have experience of Apache Flink/Apache Kafka and the ELK stack (Elasticsearch, Logstash & Kibana)
• great communicator, you know how to speak with people at all levels.
• team player, working with a globally distributed team.

Professional & Technical Skills :

• Must To Have Skills: Proficiency in Microsoft Azure Databricks.
• Experience with cloud-based data engineering and analytics platforms.
• Strong knowledge of distributed computing and big data processing frameworks.
• Ability to design scalable and high-performance data pipelines.
• Familiarity with data security and governance best practices.
• Experience in troubleshooting and optimizing complex data workflows.

Additional Information :

• The candidate should have minimum 5 years of experience in Microsoft Azure Databricks.
• This position is based at our Pune office.
• A 15 years full time education is required.