Summary
Overview
Work History
Education
Skills
Certification
Affiliations
Timeline
Generic

Gaurav Kalwani

Charlotte,NC

Summary

Senior Data Engineer with over 11 years of experience in developing and enhancing data pipelines and ETL processes. Expertise in Python, SQL, Apache Spark, and cloud technologies including GCP and AWS. Proven track record in leading teams to design scalable data architectures and deliver actionable insights. Skilled in managing comprehensive data strategies to drive performance and innovation in fast-paced environments.

Overview

12
12
years of professional experience
1
1
Certification

Work History

Sr Data Engineer

American Tire Distributors Inc
Huntersville, NC
12.2023 - Current
  • Led the migration of data from Snowflake to Google Cloud Platform (GCP), designing and implementing data marts aligned with business domains.
  • Utilized strong expertise in SQL, Python, and data engineering, employing libraries such as FastAPI, Pandas, and NumPy for data cleaning, transformation, and orchestration.
  • Engineered robust Python-based ETL/ELT pipelines using Cloud Functions, Cloud Run, Apache Airflow (GCP Composer), and DBT to extract and transform data from REST API endpoints into the BigQuery Data Warehouse.
  • Integrated Hadoop and Apache Spark to process and analyze large-scale datasets, enabling high-performance distributed data processing.
  • Employed Teradata for legacy data source integration and optimization of historical data analysis.
  • Architected and implemented CI/CD pipelines using Terraform and GitHub Actions, improving deployment speed by 40% and ensuring efficient, automated delivery across development, staging, and production environments.
  • Designed and deployed ELT pipelines processing over 1TB of data daily, leveraging DBT to automate transformations and improve data accuracy by 20%.
  • Built scalable data pipelines for both real-time streaming and batch processing workflows.
  • Demonstrated technical proficiency in data modeling, database design, and data mining techniques across modern and traditional data platforms.
  • Delivered high-impact projects ahead of schedule in agile environments, providing actionable insights to senior leadership that influenced key business decisions.

Sr Data Engineer

Torqata Data and Analytics LLC
Huntersville, NC
10.2022 - 12.2023
  • Conducted data modeling and monitored daily feeds into BigQuery from various sources using Grafana and Looker.
  • Developed Python programs using BigQuery cloud client libraries for schema modifications and data transformations.
  • Developed efficient and scalable microservices using Python and FastAPI for data extraction from external sources, enabling real-time data processing, improved data quality.
  • Generated reusable SQL queries to enhance interoperability and facilitate data searches.
  • Actively participated in code reviews, meetings, and scrum sessions to track project progress and align with business needs.
  • Enhanced data reliability and efficiency by implementing a scorecard dashboard on Looker to measure data quality, improving match rates for transaction line validations.
  • Enhanced Python code to enrich the data flow within the platform.
  • Conducted root cause analysis of data and processes to address business queries and identify areas for improvement.
  • Designed Grafana alerts and dashboards for data quality control for new customers.
  • Engaged with stakeholders and the business team to align with technical requirements effectively.

Integration/Data Engineer

Miracle Software Systems Inc
Novi, MI
02.2016 - 10.2022
  • Company Overview: Major Clients: McDonald’s, JB Hunt, HUB Group
  • Designed and implemented robust ETL pipelines using Python and PySpark, improving data extraction, transformation, and loading processes from diverse sources.
  • Experienced with tools such as BigQuery, Looker, PostgreSQL, MS SQL, Talend, Google Cloud Platform, version control tools like GitHub.
  • Proficient in JIRA, Confluence, and familiar with streaming services like Kafka and Google Pub-Sub.
  • Developed and optimized data pipelines for large-scale data processing using Hadoop HDFS to store and manage structured and unstructured data, improving accessibility and scalability.
  • Implemented complex queries and transformations in Apache Hive to analyze and extract insights from big data stored in HDFS, significantly reducing query execution times.
  • Troubleshot and resolved critical ETL issues, ensuring timely delivery of high-quality data, while optimizing pipelines for cloud environments like GCP (BigQuery, DataProc, Apache AirFlow, Cloud Functions).
  • Skilled in Business Process Development within Sterling Integrator, with expertise in EDI transaction sets.
  • Hands-on experience with various adapters like SAP Suite, LJDBC, FTP, SFTP, HTTP, and IBM Sterling File Gateway (SFG) for secure file transfers.
  • Architected and designed IBM Sterling File Gateway (SFG) and demonstrated competence in IBM Sterling Connect: Direct, UNIX/Windows environments, and Shell scripting.
  • Proficient in designing and developing B2B integration maps and components using IBM SI 5.x, encompassing ANSI X12, EDIFACT, VDA, SAP IDOC, XML, CSV, JSON and custom formats.
  • Installation and configuration of IBM Control Center for SI nodes, SFG, FTP Servers, and Connect Direct Nodes on UNIX and Windows platforms.
  • Re-architected Business Processes for scaling and optimization.
  • Designed a Trading Partner Onboarding portal and migrated 700+ partners to IBM DataPower.
  • Administered a production MFT system (Sterling Integrator-SI, Sterling File Gateway-SFG, IBM Control Center-ICC) for Windows and UNIX.
  • Served as a Technical SME for Sterling Integrator 5.x, 6.0 implementations and operations.
  • Designed and maintained Business Processes for migration to new DMaaS architecture.
  • Installed multiple IBM CD instances across client locations for seamless restaurant business operations.
  • Set up data communication protocols (FTP, HTTP, HTTPS, SFTP, AS2) and integrated SI with AWS S3, Azure, and BOX.
  • Tuned SI property files and components for optimal performance.
  • Worked with Oracle and SQL Server databases, developing SQL, PL/SQL scripts, functions, and procedures.
  • Major Clients: McDonald’s, JB Hunt, HUB Group

Jr Software Developer

Miracle Software Systems Inc
Visakhapatnam, India
09.2013 - 07.2014
  • Spearheaded the migration of maps and business processes from Gentran Unix server to Sterling Integrator, ensuring seamless transition.
  • Crafted robust test cases to validate expected and negative outcomes, guaranteeing system reliability.
  • Proficiently managed Trading Partner profiles via FTP and AS2 protocols, optimizing data exchange.
  • Led the successful CarMax EDI migration to CARQUEST GPIT using Sterling Integrator, enhancing efficiency.
  • Mastered multitasking by engaging with multiple clients simultaneously, streamlining complex business flows.

Education

Master of Science - Computer Science

Bradley University
Peoria, IL, USA
12.2015

Bachelor’s - electrical and Electronics Engineering

Jawaharlal Nehru Technological University
India
05.2013

Skills

  • ETL and data pipelines
  • SSIS and SQL Server
  • Apache Airflow and Hadoop
  • Spark and DBT
  • HIVE and Teradata
  • Dimensional modeling
  • Snowflake and BigQuery
  • Orchestration and automation
  • Terraform and CI/CD
  • GitHub and Bitbucket
  • Docker and Kubernetes
  • Python, Java, and C#
  • Shell scripting and UNIX
  • SQL and cloud platforms
  • Google Cloud Platform, AWS, and Azure
  • Tableau, Power BI, and Looker
  • Data formats: JSON, CSV, XML, Parquet
  • IBM Sterling B2B tools

Certification

  • 2 x Google Cloud Professional Data Engineer
  • Mastering DBT (Data Build Tool) - From Beginner to Pro - Udemy
  • IBM Sterling B2B Integrator V5.2, Solution Implementation
  • IBM Sterling B2Bi v5.2.6x - Developer

Affiliations

  • Wholehearted Learner – 03/2023

Timeline

Sr Data Engineer

American Tire Distributors Inc
12.2023 - Current

Sr Data Engineer

Torqata Data and Analytics LLC
10.2022 - 12.2023

Integration/Data Engineer

Miracle Software Systems Inc
02.2016 - 10.2022

Jr Software Developer

Miracle Software Systems Inc
09.2013 - 07.2014

Master of Science - Computer Science

Bradley University

Bachelor’s - electrical and Electronics Engineering

Jawaharlal Nehru Technological University

Similar Profiles

CREATE PROFILE