Summary
Overview
Work History
Education
Skills
Websites
Accomplishments
Timeline
Generic

PALLAVI BHARDWAJ

New York,NY

Summary

  • Results-driven Data Engineer with over 8 years of experience in designing developing, and optimizing data and ETL workflows for large-scale data processing.
  • Expertise in GCP migration projects utilizing BigQuery, SQL, and Python to significantly enhance data processing efficiency.
  • Experienced Google Cloud Data Engineer with expertise in designing, developing, and optimizing scalable data pipelines using Google Cloud Platform (GCP) services such as. Big Query, Cloud Dataflow, and Cloud Composer (Apache Airflow).
  • Hands-on experience in implementing CI/CD pipelines for data engineering workflows using tools such as Jenkins enabling automated deployments and testing.
  • Used JIRA and Rally to Bug/Issue tracking and project management.
  • Proven ability to lead seamless data migrations and conduct comprehensive Proof of Concepts (POCs) across diverse technologies.
  • Strong collaboration skills foster effective communication across teams, ensuring alignment on project objectives and successful outcomes.

Overview

12
12
years of professional experience

Work History

Data Engineer

Ascendion USA
08.2022 - Current

Company Overview: Client - Aetna

  • Spearheaded a large-scale GCP migration project, successfully transitioning the company's data infrastructure to the cloud
  • Worked on migrating GCP tenants from shared to their own compute projects in order to monitor expenses, utilization, and improve the estimation of their business budget.
  • Refactored existing on-premises code to Airflow DAGs, leveraging BigQuery and Python for efficient data processing
  • Developed and optimized SQL queries to extract, transform, and load data from various sources
  • Conducted numerous POCs involving FTP, Dataproc, and other technologies to evaluate their suitability for specific use cases.
  • Led the data migration effort, ensuring data integrity, security, and compliance throughout the process
  • Collaborated with cross-functional teams to ensure accurate data migration and to resolve other dependencies for smooth migration process
  • Automated the process to validate and compare tables migrated by two different tools for data migration team
  • Automated the data ingestion process using Dataproc, ingesting data from BigQuery to various databases
  • Provided training and mentorship to team members, fostering knowledge sharing and professional growth
  • Participated in agile development processes, contributing to sprint planning, stand-ups, and reviews to ensure timely delivery of data projects
  • Optimized SQL queries and database schemas for performance improvements in data retrieval operations
  • Designed, constructed, and maintained scalable data pipelines for data ingestion, cleaning, and processing using Python and SQL
  • Implemented data visualization tools like Tableau and Power BI to create dashboards and reports for business stakeholders
  • Developed Python scripts for extracting data from web services API's and loading into databases
  • Managed version control and deployment of data applications using Git, Docker, and Jenkins
  • Analyzed user requirements, designed and developed ETL processes to load enterprise data into the Data Warehouse
  • Automated ETL processes across billions of rows of data, which reduced manual workload by 29% monthly
  • Designing and developing scalable data pipelines using Google Cloud Platform (GCP) services such as Cloud Dataflow, Cloud Composer, and Apache Beam to process and manage large financial datasets efficiently.
  • Building and optimizing data storage solutions leveraging Big Query, Cloud SQL, and Cloud Spanner to ensure high-performance data retrieval and analytics for financial reporting and compliance needs.
  • Developing ETL/ELT workflows using Cloud Data Fusion, Cloud Storage, and Python, ensuring seamless data ingestion from various sources, including transactional banking systems and third-party financial services.
  • I worked with data scientists to design data structures and support machine learning (ML) models using Google AI Platform and Big Query ML.
  • Implementing CI/CD pipelines for data workflows using Cloud Build, Terraform, and GitHub Actions, automating deployment, testing, and monitoring processes to ensure consistency and reliability.
  • Developing and maintaining data pipelines using GCP services such as Cloud Dataflow, Big Query, and Cloud Pub/Sub to ensure seamless data ingestion, transformation, and storage.
  • Automating infrastructure provisioning and deployments using Terraform, Cloud Deployment Manager, and CI/CD pipelines with Cloud Build and GitHub Actions to ensure consistency and scalability.
  • Supporting machine learning and AI initiatives by preparing and transforming large datasets for model training and evaluation using Big Query, Vertex AI, and Cloud AI Platform.
  • Monitoring and troubleshooting data workflows using Cloud Logging, Cloud Monitoring, and Error Reporting, ensuring minimal downtime and quick resolution of data pipeline issues.
  • Migrating on-premises Hadoop clusters (including HDFS, YARN, and MapReduce) to Google Cloud Storag

Environment: Google Cloud Platform (GCP), Apache Beam, BigQuery, Cloud SQL, Cloud Spanner, ETL/ELT workflows, Cloud Data Fusion, Apache Kafka, Google AI Platform, CI/CD pipelines, and GitHub Actions.

Sr. Data Analyst - Datacenter Services

NIIT Technologies Pvt. Ltd (COFORGE)
Noida, India
11.2018 - 12.2019

Company Overview: Client - British Airways

  • Monitored brand mentions and reviews across various platforms, performing sentiment analysis using Python libraries (Beautiful Soup, TextBlob) for brand monitoring and reputation management
  • Created interactive Tableau dashboards to showcase sentiment analysis, identifying friction points and improving customer perception, resulting in a ~10% increase in conversion rate
  • Analyzed 3 million records of airline data using SQL to create reporting dashboards in Power BI, highlighting KPIs from various operational aspects for stakeholders
  • Worked on a pilot project to develop fraud triggers using historical data to flag high-risk transactions and identify drivers of fraud bookings
  • Led a team of 5 data analysts, compiling daily, weekly, monthly, and annual reports, and conducting ad hoc data mining for executive leadership
  • Analyzed complex datasets to identify trends and patterns

Environment: Tableau Desktop, Tableau Server, Power BI Desktop, Power BI Service, SQL Server, Python, Excel

Data Analyst | Software Developer

Premier Logic (Acquired by Alten Cal Soft labs)
Noida, India
11.2014 - 11.2018

Environment: Tableau Desktop, Python, PHP, AngularJS, Jira, SQL, Excel.

Company Overview: Client - Imarticus Learning, Jubi, DGC

  • Developed and implemented comprehensive solutions for multiple Customer Relationship Management (CRM) campaigns, facilitating seamless outreach to customers across various channels, including email, web, and mobile, tailored to meet client-specific requirements
  • Presented findings and insights through visualizations and written reports to stakeholders
  • Developed a full-stack online learning platform using PHP, MySQL, and Angular, featuring secure user authentication, course management, interactive learning modules, and gamification elements
  • Created an interactive data visualization dashboard using Tableau and Python to analyze student performance data, identify learning gaps, and provide actionable insights to educators and administrators
  • Designed and developed a learning platform with two distinct dashboards: one for the instructor and another for the students, featuring real-time data updates and interactive visualizations.
  • Automated the process of fetching student results every 30 minutes and updating the database and instructor's dashboard, ensuring timely access to up-to-date information

Research and Development Engineer

MRM Procom
Faridabad, India
02.2013 - 11.2014
  • Embedded System Design Based Projects and Electronics Hardware Based Projects: Hardware & Firmware Designing, Project Monitoring & Testing
  • Developed protocols like CAN, Modbus, and USB for microcontrollers used for communication
  • Embedded System Design Based Projects and Electronics Hardware Based Projects: Hardware & Firmware Designing, Project Monitoring & Testing

Education

Big Data Analytics - Post Graduation Diploma

Georgian College
Barrie, Canada
01.2022

PG Diploma - Embedded Systems and Design

Centre For Development of Advanced Computing
Hyderabad, India
01.2013

Bachelor of Technology - Electrical and Electronics

Maharshi Dayanand University
Rohtak, India
08.2012

Skills

Cloud Platforms & Services:

  • GCP Services: Cloud Storage, Dataproc, Bigquery, Dataflow, Cloud Composer, spanner, Notebooks, Vertex AI

Programming & Scripting

  • Languages: Python (Pandas, NumPy, PySpark), SQL, Shell Scripting
  • Frameworks & Libraries: PySpark, Spark SQL

Data Engineering & ETL Tools

  • ETL Tools: BigQuery, Apache Airflow
  • Big Data Ecosystem: Apache Spark, Hadoop (HDFS, MapReduce), Hive, Kafka

Databases:

  • Relational Databases: MySQL, MS SQL Server
  • NoSQL Databases: MongoDB

Reporting & Visualization

  • Power BI, Tableau

DevOps & CI/CD:

  • Jenkins

Tools & Utilities:

  • Visual Studio Code, PyCharm, Jupyter Notebooks, SQL Server Management Studio (SSMS)

Security & Governance:

  • Data Encryption, Data Classification and Governance

General Concepts:

  • Data Warehousing (Star Schema, Snowflake Schema), Data Modeling (Dimensional Modeling, ER Diagrams), Performance Tuning (SQL Query Optimization, Indexing), Batch Processing

Accomplishments

Client Whisperer Award, 01/2024, Awarded by Aetna for exceptional client relationship management and high satisfaction rating among over 200 nominees.

Timeline

Data Engineer

Ascendion USA
08.2022 - Current

Sr. Data Analyst - Datacenter Services

NIIT Technologies Pvt. Ltd (COFORGE)
11.2018 - 12.2019

Data Analyst | Software Developer

Premier Logic (Acquired by Alten Cal Soft labs)
11.2014 - 11.2018

Research and Development Engineer

MRM Procom
02.2013 - 11.2014

Big Data Analytics - Post Graduation Diploma

Georgian College

PG Diploma - Embedded Systems and Design

Centre For Development of Advanced Computing

Bachelor of Technology - Electrical and Electronics

Maharshi Dayanand University
PALLAVI BHARDWAJ