Kapil Shrivastava
Summary
Hands-on engineering leader with experience in building scalable web applications, datamart and datalakes. Broad knowledge of Big Data, Analytics. Passionate about implementing latest AI technologies to deliver world class products.
Results-oriented achiever with proven ability to exceed targets and drive success in fast-paced environments. Combines strategic thinking with hands-on experience to deliver impactful solutions and enhance organizational performance.
Overview
21
21
years of professional experience
1
1
Certification
Work History
Software Development Manager (Big Data Technologies)
Amazon
07.2022 - Current
- Directed and mentored multiple engineering teams, driving the end-to-end delivery of Amazon’s cloud-based data lake marketplace, which empowered secure, scalable data sharing for thousands of enterprise users.
- Defined and executed technical strategy for core platform services including cataloging, data storage, secure access, governance/compliance, data discoverability, fine grain access control, monitoring, observability, cost optimization and data quality using state-of-the-art AWS big data technologies (S3, EMR, Glue, Redshift, Athena, SageMaker, LakeHouse, DataZone, AWS lambda, SQS, Kafka, Airflow MWAA) and frameworks (Apache Spark, Scala, Java).
- Accelerated project delivery by 25% by instituting rigorous Agile methodologies, optimizing sprint planning, and fostering a culture of continuous feedback and improvement.
- Lead the team to deliver Amazon wide SageMakerUnifiedStudio, which brings centralized catalog authentication with Amazon IDC and enables GenAI use cases for multiple teams across Amazon.
- Launched platform-wide self-service analytics tools with GenAI, reducing manual data request volume for partner teams by 40% and speeding data-driven business decisions.
- Pioneered team adoption for Amazon Kiro, CLINE and internal Amazon bedrock models, prompt engineering, Amazon Q (Developer), Anthropic Claude Sonnet
- Built Agentic workflows with custom MCP servers to optimization operations and support
- Resource optimization with ML models for spark job run parameters to reduce 25% idle time and increased resource utilization.
Senior Engineering Manager(Big Data Analytics)
McGraw-Hill Education
06.2016 - 07.2022
- Leading Analytics and reporting groups data engineering teams to build spark/scala/AWS based scalable data ingestion (spark streaming) and ETL pipelines.
- Responsible for building data lake for storage and processing of terabytes of data.
- Leading the team to also build and deploy scala based microservices using scalatra/docker/AWS ECS/Terraform.
- Acting as a SME for all databricks and spark structured streaming related work.
- 13 years
Software Engineering Manager
McGraw-Hill Education
07.2014 - 07.2016
- Architecture and design of McGraw Hill's cloud based data infrastructure that can store/process millions of customers web interaction events.
- Leading/Managing the team to develop high quality, open standards based products.
- Managing development activities, providing directions to the teams.
- Responsible for complete lifecycle of products from inception to go live.
Lead Software Engineer
McGraw-Hill Education
08.2009 - 07.2014
- Leading migration of complete suite of applications from 2 data centers to AWS environment.
- Leading the team to build and deliver products that are following open technology design.
- Designed and integrated multiple systems with a generic design that is re-usable for future products.
- Successfully designed/implemented complex ETL jobs to process millions of transactions in few minutes.
Senior Software Engineer
Altech Star Inc
01.2008 - 08.2009
- Build next-generation learning tools that will be used by millions of learners and instructors using latest tech stack from J2EE
- Implemented highly transactional LMS integration using JPA/Hibernate and Spring.
- Implemented performance optimizations with caching
Senior Software Engineer
HSBC
07.2004 - 12.2007
- Java/J2EE, WebSphere,IBM DB2, MQ Series etc for building private banking product for trading, the app supported Stocks, Mutual Funds and Options.
- Built complex calculation and data processing apps.
Education
Bachelor's degree - Electronics and Telecommunications
Institute for Professional Studies
Indore, IndiaSkills
- Amazon Web Services (AWS)
- Databricks
- Artificial Intelligence (AI)
- Gen AI
Certification
- Sun Certified Java Developer
- Apache Spark 2.4
- Image Classification with Amazon Sagemaker
Honors-Awards
Best Performer Award, 2015 CODiE award for Best Postsecondary Personalized Learning Solution
Timeline
Software Development Manager (Big Data Technologies)
Amazon
07.2022 - Current
Senior Engineering Manager(Big Data Analytics)
McGraw-Hill Education
06.2016 - 07.2022
Software Engineering Manager
McGraw-Hill Education
07.2014 - 07.2016
Lead Software Engineer
McGraw-Hill Education
08.2009 - 07.2014
Senior Software Engineer
Altech Star Inc
01.2008 - 08.2009
Senior Software Engineer
HSBC
07.2004 - 12.2007
Bachelor's degree - Electronics and Telecommunications
Institute for Professional Studies