Vaishnavi Thula

• Contributed to the development and maintenance of STS Insights, a cloud-based authoritative data mart built on Google BigQuery for Schwab Technology Services. The platform integrates fine-grained process data from multiple enterprise sources — including ServiceNow/Helix (incidents, change, assets, CMDB), Jira, GitHub, Clarity, and APM — organized across four subject areas: STS Planning, STS Build, STS Run, and STS Risk. Responsibilities included designing and optimizing BigQuery stored procedures for ELT pipelines, implementing SCD (slowly changing dimension) patterns for warehouse fact and dimension tables, troubleshooting data quality issues, and enabling self-service analytics for citizen data analysts across the organization.
• Responsibilities:
• End-to-End Data Integration:
• Applies requirement analysis techniques and tools to independently manage requirements for projects.
• Assist in design and specification activities for projects, applying design principles and standards to develop software solutions.
• Build or support data pipelines and reporting solutions.
• Assist in automation processes including monitoring, provisioning and testing.
• Perform programming activities, codes, tests, debugs.
• Collaborate with cross-functional teams and stakeholders to propose innovative solutions.
• CI/CD Pipeline Automation:
• Pipeline Deployment: Established SSIS packages and CI/CD pipelines to automate the deployment process. This automation decreased manual deployment efforts by 40% and ensured consistent, error-free pipeline management across various environments.
• Version Control: Implemented version control practices to manage pipeline versions and deployment processes effectively.
• Performance Monitoring: Enabled proactive detection of performance issues and anomalies, ensuring high availability and reliability of data streaming solutions.
• Advanced Querying and Analytics:
• Query Optimization: Leveraged advanced querying and analytics, integrating data from multiple sources to provide near real-time insights. Reduced query processing times by 20% through effective query optimization and indexing strategies.
• Data Insights: Facilitated data-driven decision-making by providing actionable insights and analytics to stakeholders.
• Data Security and Compliance:
• Security Measures: Implemented robust data security measures to comply with GDPR and CCPA regulations. This included encryption-at-rest and in-transit, role-based access control (RBAC).
• Compliance Audits: Conducted regular audits to ensure adherence to data protection regulations and industry standards.
• Monitoring and Logging Solutions:
• Comprehensive Monitoring: Created comprehensive monitoring and logging solutions using Azure Monitor, Log Analytics, and Datadog. Ensured 99.9% uptime for critical ETL processes and facilitated proactive issue detection.
• Alerting: Configured advanced alerting mechanisms to quickly address and resolve system performance bottlenecks and anomalies.
• Data Governance and Discovery:
• Data Lineage & Cataloging: Implemented data governance solutions using Google Cloud Dataplex and Data Catalog to track data lineage, classify datasets, and manage metadata across GCP environments. Improved data visibility and enabled efficient data discovery for analytics teams.
• Governance Framework: Designed and enforced data governance standards to ensure data integrity, quality, and compliance across data pipelines and storage layers (BigQuery, Cloud Storage). Strengthened data stewardship practices organization-wide.
• Development, Version Control & Deployment:
• Code Management & Collaboration: Leveraged GitHub and Bitbucket for version control, branching strategies, and collaborative development of data pipelines.
• Development Environment: Built and maintained data engineering solutions using Microsoft Visual Studio and GitHub Copilot, accelerating development through AI-assisted coding and standardized practices.
• Deployment Efficiency: Streamlined CI/CD processes using repository-integrated workflows, improving deployment consistency, reducing manual effort, and enhancing overall system reliability in GCP environments.
• Additional Skills and Technologies:
• Data Processing Tools: Utilized tools such as Tableau for data visualization and reporting, enhancing the presentation and analysis of complex datasets.
• Environment: Python, SQL, Tableau, Data Lake, Data Storage, Data Factory, Shell scripting, Linux, GitHub,BitBucket, MySQL, Jira, Zephyr,and Agile Methodologies.

Responsibilities:

• Data Integration and Pipeline Development: I integrated various source systems such as Azure SQL Server, Azure Data Lake Storage (ADLS), Blob Storage, and Rest APIs into robust data pipelines using Azure Data Factory (ADF). This facilitated the seamless ingestion and transformation of structured and unstructured data across multiple systems. I worked on ingesting data from a variety of sources like SAP HANA, Cloudera HDFS, and SFTP Servers into Azure Data Lake Storage Gen 2, ensuring efficient ETL processes.
• Building Full-Load Azure Data Factory Pipelines: Developed and deployed full-load pipelines in ADF to migrate data from legacy systems into the cloud. These pipelines extracted data from disparate sources and ensured data integrity and consistency during migration. I played a key role in creating pipelines to facilitate real-time and batch processing of data for various reporting and analytics purposes. Developed scalable ETL pipelines using Airflow, Python, and BigQuery on GCP, optimizing for large-scale data lakes, and achieving significant cost savings through efficient data partitioning and transformation strategies.
• Azure Databricks and Spark for Data Transformation: Leveraged Azure Databricks and its distributed computing capabilities to develop and implement data transformation workflows. I built Spark-based jobs in Databricks for complex data transformations, ensuring scalability and efficiency. These transformations were essential for downstream analytics and reporting systems, enabling data-driven insights for stakeholders.
• Data Warehousing and Snowflake Implementation: Designed scalable data models using Azure SQL Data Warehouse and Snowflake. This involved building schemas, tables, views, and other data structures to handle large datasets efficiently. I optimized Snowflake’s micro-partition architecture to enhance query performance, reduced processing times, and streamlined reporting operations. I utilized Snowflake’s features such as Time Travel to enable historical data analysis, ensuring compliance with regulatory requirements and aiding in point-in-time data recovery.
• Real-Time Data Streaming and Azure Functions: Implemented data streaming solutions using Azure Event Hubs and Azure Functions to enable real-time data processing. I designed pipelines that allowed data to flow seamlessly from Event Hubs into Snowflake, where it was processed and stored for real-time analytics. These streaming solutions were crucial for enabling real-time business insights, particularly in supply chain and inventory management scenarios.
• Data Security and Compliance: Ensured data security and compliance by developing data compression and encryption techniques within Azure Blob Storage and Azure Data Lake Storage. I implemented end-to-end data lineage and cataloging solutions using Azure Purview and Apache Atlas, enabling complete transparency and traceability for compliance audits. This was especially critical in safeguarding sensitive customer and transactional data.
• Orchestration and Automation of Workflows: Automated complex data workflows by integrating Azure Data Factory with Azure Logic Apps. This integration allowed me to automate event-driven triggers that executed specific actions, such as notifying business users or triggering downstream processes. I also configured alerts and pipeline triggers to ensure continuous monitoring of data pipelines, minimizing downtime and ensuring efficient data processing.
• Data Governance and Quality Management: Implemented data governance frameworks within Azure Data Factory and Snowflake, ensuring strict data quality checks and validation mechanisms were in place. By using tools like Azure Monitor and custom monitoring solutions, I was able to proactively identify data inconsistencies and resolve performance bottlenecks. This improved the overall reliability and accuracy of data flowing through the pipelines.
• Collaboration with Business and Technical Teams: Worked closely with cross-functional teams, including data scientists, business intelligence analysts, and business stakeholders, to gather data requirements and deliver tailored solutions. This collaboration resulted in the development of data pipelines that aligned with the business objectives and provided actionable insights for decision-making.
• Self-Service Analytics Enablement: Enabled business users to perform self-service analytics by integrating Snowflake with Power BI and Azure Analysis Services. I developed interactive dashboards and reports in Power BI and Tableau, providing visual insights into key metrics such as sales performance, customer behavior, and inventory levels. These tools allowed non-technical users to explore and analyze data with ease, improving operational decision-making.
• Monitoring and Troubleshooting: Developed custom monitoring and alerting mechanisms using Snowflake Query Performance Monitoring (QPM) and Azure Monitor. These tools allowed me to proactively detect performance issues, optimize query performance, and minimize system downtimes. I also fine-tuned Spark jobs within Azure Databricks, improving overall pipeline performance by optimizing resource allocation and implementing caching techniques.
• Advanced Analytics and Machine Learning Workflows: Worked on machine learning workflows that involved data preprocessing, feature extraction, and model deployment using Azure Machine Learning and Snowflake. These models provided predictive insights into business operations, such as demand forecasting and customer churn predictions, empowering with data-driven decision-making capabilities.
• Version Control and DevOps Implementation: Used Azure DevOps and GIT for version control, enabling a streamlined CI/CD process for deploying and maintaining data pipelines. I implemented infrastructure-as-code solutions and automated testing, reducing the time required to deploy and update pipelines and ensuring that changes were tracked and managed efficiently.
• Documenting and Transitioning Data Platforms: Documented all processes involved in transitioning data platform from IBM DataStage to Azure, ensuring that stakeholders had a clear understanding of the changes. My documentation provided comprehensive insights into the technical architecture, data flow, and performance enhancements achieved during the transition, helping the team to maintain the system effectively post-implementation.
• Key Technologies and Tools: Cloud & Data Platforms: Azure Data Factory, Azure Data Lake, Snowflake, Azure SQL Data Warehouse, Blob Storage.
• Data Engineering: Azure Databricks, Spark, Python, Scala, PySpark

• The Fraud Detection project aims to enhance the security and integrity of financial transactions by automatically identifying and flagging suspicious activities. This system integrates with financial transaction systems to analyze transaction records, identify potential fraud, and provide actionable insights. The solution involves storing comprehensive data from user accounts and credit card transactions to detect and address fraudulent activities effectively.
• Responsibilities:
• Data Pipeline Design and Development:
• Pipeline Architecture: Designed and developed data pipeline jobs to ingest data from various source systems. Upgraded, configured, and maintained Hadoop infrastructures, including Pig, Hive, and HBase.
• Data Ingestion: Utilized Sqoop, Flume, MapReduce, and Kafka to handle data ingestion, transformation, and processing. Enabled advanced analytics by efficiently managing customer behavioral data.
• Data Processing and Transformation:
• Spark and Scala: Conducted data aggregation and analysis using Apache Spark, Scala, and Hive, leading to enhanced business insights through advanced data processing capabilities.
• Python Scripting: Developed Python scripts to facilitate seamless data transfer and integration between different data sources, demonstrating proficiency in Python for effective data management.
• Real-Time Data Handling:
• Spark Streaming: Developed Spark Streaming applications to process both static and streaming data from various sources, such as SQL Server, EDW, and OLTP data stores, enabling real-time data processing and analytics.
• Workflow Automation:
• Oozie Workflow: Designed and implemented automated workflows using Oozie to streamline data loading into HDFS, pre-processing, analysis, and classifier training tasks. Utilized MapReduce, Pig, Zookeeper, and Hive jobs to achieve seamless data operations.
• Big Data Ecosystem Utilization:
• Ecosystem Tools: Leveraged Hadoop, Spark, and Cloudera for loading and transforming large datasets, including structured, semi-structured, and unstructured data.
• HBase and Hive Integration: Optimized HBase tables for efficient querying and integrated HBase with Hive to accelerate data retrieval and performance.
• Data Integration and Migration:
• Data Migration: Executed successful data migration projects from Oracle RDBMS to Hadoop using Sqoop, enhancing data management and processing capabilities.
• Data Pipelines: Developed and maintained data pipelines, including data ingestion from Kafka and persistence in HBase, ensuring efficient and reliable data flow.
• Visualization and Reporting:
• Tableau and Power BI: Designed and implemented visually compelling visualizations using Tableau and Power BI, transforming raw data into insightful charts and graphs for strategic decision-making.
• System Monitoring and Optimization:
• Ambari Management: Utilized Apache Ambari for efficient management and monitoring of Hadoop clusters, enabling optimal resource utilization and troubleshooting.
• Performance Optimization: Improved Spark performance by leveraging Spark Context, Spark Sessions, Spark-SQL, DataFrames, and Spark YARN for enhanced algorithm efficiency.
• Automation and Scripting:
• YAML Automation: Implemented YAML script-based automation for deployments, accelerating build and release processes to achieve faster and more efficient workflows.
• RESTful Services: Developed middleware using RESTful web services and Python scripts to parse JSON documents and load data into databases.
• Additional Skills and Technologies:
• Nifi Workflows: Developed Nifi workflows for data extraction from REST API servers, data lakes, and SFTP servers, and sent data to Kafka brokers.
• Kafka Integration: Created Kafka producers and developed Spark streaming applications to handle real-time data processing. Monitored consumer lag within Apache Kafka clusters.
• Hive and Spark Integration: Created and maintained Hive tables, executed Hive queries, and developed Spark scripts using Scala and Spark SQL to optimize data processing.
• Agile Methodology: Worked extensively in Agile iterative sessions to create a Hadoop Data Lake and generate actionable insights from complex datasets for various application teams.
• Version Control: Utilized GIT for version control, ensuring effective code management and change tracking.
• Environment: Azure Databricks, Data Factory, Logic Apps, Snowflake, Functional App, Snowflake, MS SQL, Oracle, HDFS, MapReduce, YARN, Spark, Hive, SQL, Python, Scala, PySpark, Shell Scripting, Kafka, Power BI.
• Analyzed data trends to support strategic decision-making processes.
• Developed comprehensive reports to communicate insights and recommendations effectively.

• Worked on employee payroll processing using Oracle E-Business Suite (EBS), supporting accurate and timely payroll operations. Developed and customized reports using Oracle Reports Builder and XML Publisher (BI Publisher) to meet business and regulatory requirements. Customized workflows and forms based on business needs, enhancing system functionality and user experience. Provided critical production support by troubleshooting and resolving issues during month-end, quarter-end, and year-end cycles, ensuring seamless payroll processing. Additionally, created data reports and dashboards using Tableau, enabling stakeholders to analyze data effectively and support decision-making.
• Responsibilities:

• Team Management & Functional Support: Managed internal teams, implementation partners, and vendors to ensure smooth and effective payroll operations. Provided functional support to enhance business process efficiency.

• Payroll Accuracy: Reviewed, analyzed, and verified payroll reports for accuracy. Made necessary adjustments through established procedures.

• Data Management: Captured and processed employee payroll information, including salary changes, terminations, and personal details on a monthly basis.

• Issue Resolution: Identified and resolved issues related to incorrect taxation, data discrepancies in various financial reports (like statements of earnings, Form 16, and provisional pay slips).

• Annual Activities: Ensured timely completion of annual financial activities, including provisional pay slip release, investment information updates, and Form 16 preparation.

• Budget Analysis: Analyzed and tested annual budget changes, ensuring timely application in the system to avoid discrepancies in payments and tax calculations.

• Business Interaction: Engaged with business users to develop enhancements, provide issue status updates, and resolve incidents promptly.

• Application Support: Handled support tickets and application issues, ensuring effective resolution for end-users.

• Documentation: Created and updated process documents regularly to maintain accurate and current procedural information.

• System Administration: Customized menus, added concurrent requests, and registered custom applications in System Administrator responsibilities.

• Defect Analysis: Analyzed defects logged by clients, debugged issues in various financial reports using BI publisher reports.

• Data Analysis & Reporting: Wrote queries for data analysis and validation, generated reports, and developed data modification scripts to meet business requirements. Modified various RTF reports. Made code changes to reports using BI Publisher.

• Requirements Documentation: Prepared business understanding documents and Functional Requirement Specifications (FRS) for new enhancements.

• Code Development & Testing: Involved in code development, impact analysis, unit test case preparation, and unit testing. Participated in requirement gathering, designing, and unit testing.

• Enhancements & Personalization: Developed several enhancements to business processes and personalized seeded and custom OA Framework pages to hide fields and buttons.

• Data Conversion: Worked closely with business users and product architects to perform data extracts for conversions using HCM Data Loader (HDL), including Workforce Structure Data, Worker Data, Work Relationship Data, Salary, Jobs, and Locations.

• Oracle HCM Cloud Management: Managed Oracle HCM Cloud configurations, including data roles, security profiles, job roles, application roles, and duty roles.

• Tax Rate Updates: Managed the Vertex update process, including the monthly tax rate upgrades.