Krunal Shah
Cumming,GA
Summary
- 14+ years of experience in designing and managing scalable data pipelines and cloud-based architectures.
- Proven leader in building and mentoring high-performing data engineering teams, optimizing data workflows, and improving data accuracy and reliability across complex environments.
- Skilled in collaborating with cross-functional teams to align data strategies with business objectives and deliver impactful solutions.
Overview
13
13
years of professional experience
1
1
Certification
Work History
Data Engineering Manager
Arkatechture
03.2024 - Current
- Led a cross-functional team of data engineers and data analysts to deliver scalable data lake and data visualization solutions for community banks and credit unions.
- Architected and implemented AWS-based data pipelines using S3, SNS, SQS, Lambda, ECS, Glue, and Athena, improving data reliability and scalability by 40%.
- Designed a Snowflake-based data warehouse that reduced storage costs by 25% and improved data processing speed by 40%, directly contributing to increased operational efficiency.
- Enhanced ETL processes with Python and automated data transformation tasks, reducing data processing times and providing faster insights for key business decisions.
- Mentored and managed a team of data engineers, fostering a high-performance culture through skill development, best practices in data engineering, and timely project delivery.
- Implemented Apache Airflow for ETL orchestration, improving workflow efficiency and ensuring timely data availability, which supported critical business operations.
- Advised senior stakeholders on data architecture and technology strategy, aligning technical solutions with broader business goals and contributing to long-term success.
Data Engineering Manager
The Weather Company an IBM Business
03.2020 - 02.2024
- Led a team of data engineers and DevOps specialists in the development of a data pipeline processing over 1 billion clickstream events daily, driving data-driven insights and enhancing business intelligence.
- Built and managed a data lake with over 1PB of data, which enabled data-driven decision-making for critical business initiatives.
- Collaborated with architects to design AWS-based data pipelines, using services such as MSK, Lambda, Glue, Athena, and Redshift, optimizing data handling and reducing latency in data processing.
- Implemented robust data quality checks, reducing data errors by 80% and ensuring reliable data outputs for business analytics.
- Managed the deployment of CI/CD pipelines using Jenkins and GitHub Actions, improving deployment speed and reducing downtime by 25%.
- Orchestrated data workflows with Apache Airflow, ensuring efficient ETL processes and timely data delivery for business reporting.
- Served as Scrum Master for data engineering projects, ensuring adherence to Agile methodologies and managing the full project lifecycle from requirements gathering through to production deployment.
- Provided leadership and mentorship to data engineers and DevOps team members, driving team collaboration, growth, and consistent delivery of high-quality solutions.
Sr DataOps Engineer
Weather Co., a subs. of Watson & Cloud Platform business unit of IBM
02.2016 - 02.2020
- Led the team of data engineers and systems engineer to migrate on premise data warehouse to AWS
- Architected and administered both public and private cloud infrastructure on AWS, encompassing VPC, EC2, S3, CloudFront, EBS, RDS, Direct Connect, Route 53, CloudWatch, CloudTrail, and IAM
- Directed the targeted deployment of AWS infrastructure across development, quality assurance, and production environments, ensuring consistency and reliability through Terraform code
- Implemented automated scripts in UNIX Shell to bootstrap EC2 instance infrastructure, streamlining deployment processes
- Executed comprehensive security measures provided by AWS, employing key IAM concepts to enhance the overall security posture of the infrastructure.
Sr ETL Engineer
The Weather Channel
09.2011 - 01.2016
- Led the initiative to construct a data warehouse supporting revenue reporting, ad tech analysis, and invoice generation
- Collaborated with internal stakeholders to gather and understand project requirements, subsequently translating them into concise technical documentation for implementation
- Designed a comprehensive data model using Erwin to establish a robust foundation for data structures
- Executed data modeling tasks for various data domains, proficiently applying Kimball Data Modeling methodologies
- Designed and implemented efficient ETL pipelines using Informatica PowerCenter to integrate and transform diverse data sources, including Oracle DB, MySQL DB, Salesforce, CSV files, and Excel files
- Seamlessly loaded the transformed data into the SQL Server-hosted Data Warehouse
- Ensured uninterrupted production support, recognizing the critical reliance of our customers on the timely availability of data for their daily reports.
Education
Master of Science - Electrical Engineering
Binghamton University
New York05.2010
Bachelor of Science - Electronics And Telecommunication Engineering
Mumbai University
India05.2007
Skills
Data Engineering & Cloud Technologies:
- AWS (S3, Lambda, Redshift, Glue, Athena, EC2), Azure, Snowflake, Databricks
Data Pipelines & Integration:
- Apache Airflow, Informatica PowerCenter, Talend, MSK, SQS, SNS
Programming & Scripting Languages:
- Python, SQL, UNIX Shell, XML, HTML, CSS
Databases:
- SQL Server, MySQL, Oracle, Redshift, MongoDB, Cassandra
DevOps & Automation:
- CI/CD (Jenkins, GitHub Actions), Terraform, Docker, Kubernetes, Git
Data Analytics & Visualization:
- Tableau, Power BI
Leadership & Management:
- Agile/Scrum, Team Leadership, Cross-Functional Collaboration, Project Management
Certification
AWS Certified Solutions Architect
Timeline
Data Engineering Manager
Arkatechture
03.2024 - Current
Data Engineering Manager
The Weather Company an IBM Business
03.2020 - 02.2024
Sr DataOps Engineer
Weather Co., a subs. of Watson & Cloud Platform business unit of IBM
02.2016 - 02.2020
Sr ETL Engineer
The Weather Channel
09.2011 - 01.2016
Master of Science - Electrical Engineering
Binghamton University
Bachelor of Science - Electronics And Telecommunication Engineering
Mumbai University
Similar Profiles
Davey HyamsDavey Hyams
House Mover at Smooth Moves Pro LLC.House Mover at Smooth Moves Pro LLC.
TABITHA AMESTABITHA AMES
ADMINISTRATIVE ASSISTANT at First Command Financial PlanningADMINISTRATIVE ASSISTANT at First Command Financial Planning
LEXI ZORNLEXI ZORN
Sr. IT Operations Manager at Soules FoodsSr. IT Operations Manager at Soules Foods
GABE NORRIEGABE NORRIE
Duckweed Research Student at Carleton UniversityDuckweed Research Student at Carleton University
Kishore Babu JangamKishore Babu Jangam
Data Engineering Manager at MicronData Engineering Manager at Micron