Custom Software Engineer - Data Pipeline and Cloud Solutions Developer

Accenture India Private Limited; Bengaluru; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Accenture India Private Limited in Bengaluru seeks a Custom Software Engineer to design, code, and enhance data pipelines and scalable software solutions using AWS Glue, Python, and PySpark. You will contribute to building reliable, optimized data processing workflows for large datasets in a fast-paced enterprise setting.

You will work in an agile environment to deliver ETL/ELT pipelines, data lake components, governance features, and performance improvements, collaborating with data platform,

Qualifications

• Must have AWS Glue, Python, PySpark.
• Experience with ETL and data integration.
• Familiarity with agile software development life cycle.

Responsibilities

• Build & Maintain Data Pipelines: Develop scalable ETL/ELT workflows using Python and AWS Glue (Jobs/Workflows), Lambda, and Step Functions where applicable.
• Data Lake Engineering: Design and optimize S3-based data lakes with robust metadata strategies using AWS Glue Data Catalog.
• Security & Governance: Implement AWS IAM policies/roles, fine-grained access controls for S3, KMS-based encryption, and audit-ready logging (CloudTrail/CloudWatch).
• Automation & CI/CD: Familiarity with Git, code reviews, and automated testing for data code. Package and deploy jobs/functions, maintain unit/integration tests, and set up CI/CD pipelines.
• Serverless & Event-Driven: Build event-driven data processing with S3 events, Lambda triggers optimize for cost, concurrency, and resilience.
• Data Quality & Observability: Implement validation, schema enforcement (e.g., Glue schemas), logging/monitoring, and alerting for pipelines and jobs.
• Collaboration: Work closely with data platform, analytics, and governance teams to ensure reliability, scalability, and compliance.
• (If Databricks): Develop notebooks/jobs, Delta Lake optimization (Z-ordering, OPTIMIZE/VACUUM), cluster/pool usage, and job orchestration.

Skills

AWS Glue

Python

PySpark

Education

15 years full time education

Job description

Custom Software Engineer Project Role : Custom Software Engineer Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. Must have skills : AWS Glue Good to have skills : Python (Programming Language), PySpark Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary: As a Custom Software Engineer, you will engage in the development of custom software solutions that are designed to meet specific business needs. Your typical day will involve collaborating with cross-functional teams to design, code, and enhance various components across systems or applications. You will utilize modern frameworks and agile practices to ensure the delivery of scalable and high-performing solutions, while also addressing any challenges that arise during the development process. Your role will require a proactive approach to problem-solving and a commitment to continuous improvement in software development practices.

Roles & Responsibilities:

• Build & Maintain Data Pipelines: Develop scalable, reliable ETL/ELT workflows using Python and AWS Glue (Jobs/Workflows), Lambda, and Step Functions where applicable.
• Data Lake Engineering: Design and optimize S3-based data lakes (partitioning, lifecycle policies, storage classes) with robust metadata strategies using AWS Glue Data Catalog.
• Security & Governance: Implement AWS IAM policies/roles, fine-grained access controls for S3, KMS-based encryption, and audit-ready logging (CloudTrail/CloudWatch).
• Automation & CI/CD: Familiarity with Git, code reviews, and automated testing for data code. Package and deploy jobs/functions, maintain unit/integration tests, and set up CI/CD pipelines.
• Serverless & Event-Driven: Build event-driven data processing with S3 events, Lambda triggers optimize for cost, concurrency, and resilience.
• Data Quality & Observability: Implement validation, schema enforcement (e.g., Glue schemas), logging/monitoring, and alerting for pipelines and jobs.
• Collaboration: Work closely with data platform, analytics, and governance teams to ensure reliability, scalability, and compliance.
• (If Databricks): Develop notebooks/jobs, Delta Lake optimization (Z-ordering, OPTIMIZE/VACUUM), cluster/pool usage, and job orchestration.

Professional & Technical Skills:

• - Must Have Skills: Proficiency in AWS Glue, Python (Programming Language), PySpark.
• - Strong understanding of data integration and ETL processes.
• - Experience with cloud computing services and architecture.
• - Familiarity with agile methodologies and software development life cycle.

Additional Information:

• - The candidate should have minimum 7.5 years of experience in AWS Glue, Python (Programming Language), PySpark.
• - This position is based at our Bengaluru office.
• - A 15 years full time education is required.