Amudhan Shanmugam
Senior Engineer at American Express
Dallas,Texas
Summary
Senior Software Engineer with 18+ years of experience building large scale distributed data platforms, real-time streaming systems, and cloud-native analytics solutions. Expertise in Kafka, Spark, PySpark,AWS (S3, EMR, EC2, Glue), GCP (Pub/Sub, GCS, BigQuery, Bigtable), Microsoft Azure, Scala, and reactive Microservices for high-volume event-driven architectures. Proven track record designing workflow orchestration engines, AST-based rule evaluation frameworks, and scalable execution pipelines aligned with modern AI/ML feature processing. Focused on platform reliability, resiliency, high-availability infrastructure, and AI-assisted engineering to accelerate development and operations.
Overview
18
18
years of professional experience
Work History
Senior Engineer
American Express
Dallas, TX
05.2023 - Current
- Designed and developed core platform components for a large-scale data streaming and workflow orchestration platform used to onboard and process enterprise event data.
- Designed and implemented an Abstract Syntax Tree (AST) compiler that converts domain-specific expressions into executable JSON-based processing logic, eliminating manual AST creation and reducing engineering effort by more than 80%.
- Architected a workflow compiler that transforms user-defined JSON workflows into processor-ready execution plans for dynamic orchestration.
- Architected an async processor using Scala, ZIO framework that reads metadata and configuration from the control plane PostgreSQL database and propagates it to the execution plane database, ensuring consistent and reliable configuration deployment.
- Built scalable scheduling and execution pipelines using Scala and ZIO.
- Developed automated remediation jobs using AI-assisted tooling to identify and apply security vulnerability fixes across more than 180 repositories.
- Created an internal vulnerability management application in one week using AI-assisted development techniques to automate tracking and remediation workflows.
- Deployed and managed containerized micro-services on Kubernetes (K8s), leveraging Helm charts, resource quotas, and rolling update strategies to achieve zero-downtime deployments and improve service scalability across distributed workloads.
- Leveraged AI tools to accelerate software development, debugging, code generation, and operational automation.
- Contributed to platform reliability, developer productivity, and security compliance across a large engineering organization.
- Technologies: Scala, ZIO, Apache Kafka, PostgreSQL, Kubernetes (K8s), Prometheus, Grafana, OpenTelemetry, Distributed Tracing, AI-assisted development, AI-powered Automation
Senior Software Engineer
Walmart Global Tech
Dallas, TX
02.2023 - 04.2023
Senior Software Engineer
Microsoft
Dallas, TX
06.2022 - 01.2023
- Designed and implemented enterprise-scale cloud analytics platforms on Microsoft Azure.
- Built end-to-end data ingestion, transformation, aggregation, and reporting pipelines for structured and semi-structured data.
- Developed scalable ETL solutions using PySpark and Azure Synapse Analytics.
- Automated infrastructure provisioning using Terraform and Bicep.
- Processed high-volume event-stream and enterprise data for analytics and operational reporting.
- Technologies: Azure Synapse Analytics, Python, PySpark, Terraform, Bicep, Linux
Technical Lead Engineer
Securonix
Dallas, TX
06.2020 - 06.2022
- Led architecture and development of large-scale cybersecurity analytics platforms processing billions of security events per day.
- Designed high-throughput streaming pipelines using Apache Kafka, Apache Spark, and Apache Flink.
- Built reusable analytics components on AWS EMR, reducing anomaly detection latency by 70%.
- Drove architectural decisions, design reviews, and technical mentoring to deliver scalable, fault-tolerant systems.
- Instrumented distributed services with OpenTelemetry for distributed tracing and metrics collection.
- Built Prometheus alerting rules and Grafana dashboards to monitor throughput, latency, and error rates reducing mean time to detect (MTTD) production incidents by over 40%.
- Engineered high-throughput Scala-based Spark jobs for real-time security event correlation and aggregation, leveraging Spark's Dataset API and functional programming paradigms (immutability, higher-order functions) to improve processing efficiency and reduce boilerplate across the analytics pipeline.
- Technologies: Spark, Flink, Kafka, AWS, Amazon EMR, Spark Structured Streaming, Linux
Senior Software Engineer
Walmart Global Tech
San Jose, CA
05.2018 - 06.2020
- Architected reactive microservices and high-throughput data integration systems for low-latency processing.
- Developed non-blocking services using Spring Boot, Spring Cloud, and RxJava.
- Implemented asynchronous workflows that improved scalability, resiliency, and maintainability of distributed applications.
- Technologies: RxJava, Spring Boot, Spring Cloud
Lead Software Engineer
Apple
San Jose, CA
05.2016 - 04.2018
- Designed and developed mission-critical streaming and search integration systems.
- Architected a multi-threaded pipeline to replicate data from Oracle Database to Elasticsearch.
- Improved processing efficiency by 40% through optimized concurrency and resource management.
- Implemented checkpoint-based recovery, health monitoring, and fault-tolerant handling for failures involving Kafka, Elasticsearch, Apache ZooKeeper, and Oracle Database.
- Technologies: Java, Kafka, Elasticsearch, Oracle Database
Java Tech Lead
General Motors
Phoenix, AZ
10.2014 - 04.2016
- Led technical design discussions, demos, and developer meetings to remove blockers and ensure timely delivery.
- Mentored engineers and coordinated development activities across the team.
- Implemented Enterprise Integration Patterns using Apache Camel.
- Developed high-concurrency applications using Java synchronization and concurrent collections.
- Technologies: Java, Apache Camel, Multithreading
Senior Software Engineer
InComm Payments
Chennai
06.2013 - 09.2014
- Developed enterprise applications using Spring MVC, Spring DAO, Hibernate, and Oracle.
- Designed persistence layers, POJO mappings, and CRUD operations.
- Built monitoring and alerting features for throughput, latency, and JVM/OS metrics.
- Integrated automated email notifications for operational monitoring.
- Technologies: Spring MVC, Hibernate, Oracle, Monitoring
System Analyst
Ford Motors
Chennai
01.2012 - 05.2013
- Participated in the full software development lifecycle, including prototyping, proof of concept, design, implementation, testing, and maintenance.
- Developed web applications for business reporting, system monitoring, and troubleshooting using J2EE and Apache Tomcat.
- Technologies: Java, J2EE, Tomcat
Software Engineer
Verizon Wireless
Chennai
10.2010 - 09.2011
- Designed and developed RESTful APIs for multiple business modules.
- Built enterprise applications using Spring, Hibernate, and Oracle.
- Contributed to backend service design and database integration.
- Technologies: Spring, Hibernate, Oracle, REST APIs
Software Engineer
HDFC Bank
Chennai
07.2008 - 10.2010
- Developed web applications using Spring MVC architecture.
- Managed application configuration, deployment, and release automation with Maven and Tomcat.
- Built and deployed Java/J2EE applications across enterprise application servers.
- Technologies: Spring MVC, Maven, Tomcat, Java/J2EE
Education
Bachelor's Degree - Electrical and Electronics Engineering
Anna University
01.2008
Skills
- Spark
- PySpark
- Apache Flink
- Apache Kafka
- Kafka Streams
- KSQL
- ETL / ELT
- Cloud-Native Development
- Distributed Systems
- Real-Time Streaming
- Data Analytics
- Data Modeling
- AWS (S3, EMR, EC2, Glue)
- GCP (Pub/Sub, GCS, BigQuery)
- Azure (Synapse, Data Explorer)
- Java
- Scala
- Python
- ZIO
- Spring Boot
- FastAPI
- REST APIs
- Microservices
- MySQL
- PostgreSQL
- NoSQL
- Cassandra
- Redis
- Elasticsearch
- Google Bigtable
- Terraform
- Bicep
- Docker
- Kubernetes (K8s)
- Helm
- Infrastructure as Code (IaC)
- Prometheus
- Grafana
- OpenTelemetry
- Distributed Tracing
- Git
- GitHub Actions
- Jenkins
- JUnit
- Linux
- CI/CD Pipelines
- LLM-Assisted Development
- AI-Powered Automation
Websites
Timeline
Senior Engineer
American Express
05.2023 - Current
Senior Software Engineer
Walmart Global Tech
02.2023 - 04.2023
Senior Software Engineer
Microsoft
06.2022 - 01.2023
Technical Lead Engineer
Securonix
06.2020 - 06.2022
Senior Software Engineer
Walmart Global Tech
05.2018 - 06.2020
Lead Software Engineer
Apple
05.2016 - 04.2018
Java Tech Lead
General Motors
10.2014 - 04.2016
Senior Software Engineer
InComm Payments
06.2013 - 09.2014
System Analyst
Ford Motors
01.2012 - 05.2013
Software Engineer
Verizon Wireless
10.2010 - 09.2011
Software Engineer
HDFC Bank
07.2008 - 10.2010
Bachelor's Degree - Electrical and Electronics Engineering
Anna University