Sai Priya
Wilmington
Summary
Data engineering professional with over five years of experience in Big Data technologies, ETL pipelines, and cloud platforms such as AWS and Azure. Expertise in developing high-performance ETL processes using PySpark, Apache Spark, and Hive to ensure reliable data transformation for analytical needs across Data Lakes and Data Warehouses. Proven ability to design and implement scalable data ingestion frameworks leveraging Kafka and Spark Streaming, combined with a strong foundation in SQL and advanced query optimization techniques for diverse database environments. Committed to enhancing operational efficiency through automated workflows and data validation frameworks while maintaining alignment with industry best practices in data governance and compliance.
Overview
6
6
years of professional experience
Work History
Data Engineer
TD Bank
Wilmington, DE
08.2024 - Current
- Designed ETL pipelines in PySpark and Hive, loading structured/semi-structured datasets into partitioned formats (Parquet/ORC).
- Migrated on-prem Hadoop workloads to AWS EMR, improving performance and lowering infrastructure costs.
- Automated data ingestion workflows using Airflow, handling dependencies and ensuring fault tolerance.
- Created risk dashboards in Tableau and Power BI, enabling executives to track exposure and regulatory KPIs.
- Implemented data quality rules and validation checks with SQL/Python, ensuring compliance with regulatory requirements (Basel, SOX).
- Worked with Tableau and Power BI teams to build interactive risk dashboards, enabling real-time tracking of exposures and stress-test results.
- Built incremental and CDC-based ingestion frameworks, reducing data load time for large datasets.
- Configured IAM roles, encryption, and access controls to safeguard sensitive risk data.
- Enhanced CI/CD pipelines for automated deployments, improving developer productivity and consistency.
- Created alerting mechanisms for data pipeline failures and anomalies, ensuring high availability and minimal downtime.
- Participated in Agile ceremonies (sprint planning, stand-ups, retrospectives), delivering features iteratively and collaborating with cross-functional teams.
- Provided production support for mission-critical pipelines, troubleshooting failures and optimizing system performance under tight SLAs.
Data Engineer
Blue Cross and Blue Shield of Minnesota
Hyderabad, India
12.2021 - 08.2023
- Built data pipelines using Azure Data Factory, Databricks, and PySpark, enabling seamless data integration from multiple healthcare systems into Azure Data Lake for analytics and reporting.
- Developed Spark-based ETL jobs to transform large-scale healthcare claims data, ensuring accuracy, compliance, and performance for downstream BI and actuarial analysis.
- Implemented real-time streaming using Kafka and Spark Streaming, processing patient records and claims data to support real-time dashboards and operational reporting.
- Designed data models using Erwin and Power Designer, ensuring scalability, normalization, and optimized query performance for healthcare analytics systems.
- Developed Python scripts for ETL automation, API integrations, and data validation, improving operational efficiency and reducing manual errors.
- Worked with Hive and SQL-based transformations for historical claims analysis, leveraging partitioning and bucketing strategies for performance tuning.
- Integrated pipelines with Power BI dashboards for healthcare utilization metrics, providing executives with actionable insights for decision-making.
- Implemented HIPAA-compliant data security measures including masking, encryption, and access control, ensuring patient data privacy and regulatory compliance.
- Migrated large-scale on-prem Hadoop workloads to Azure Synapse and Databricks, improving scalability and reducing operational costs.
- Used Apache Oozie and Airflow for workflow automation, improving the reliability of ETL jobs and reducing dependency on manual processes.
- Conducted performance tuning of Spark jobs, optimizing memory usage, partitioning strategies, and parallelism settings for high-throughput healthcare workloads.
Data Engineer
Home Depot
Hyderabad, India
01.2020 - 12.2021
- Designed and implemented data pipelines using Hadoop, Hive, and Spark, enabling ingestion and processing of retail transactions for large-scale analytics and customer insights.
- Developed ETL jobs using PySpark and Sqoop to extract and transform data from multiple sources into Hadoop HDFS and Hive tables, ensuring accuracy and performance.
- Created partitioned and bucketed Hive tables for optimized querying, significantly improving the performance of large analytical workloads for sales and product data.
- Integrated Hadoop-based data platform with Tableau dashboards, enabling business users to monitor real-time KPIs and operational metrics across stores.
- Implemented error handling, retries, and fault-tolerant mechanisms in Spark jobs to ensure reliability in production environments.
- Automated data validation and reconciliation processes using Python, reducing manual intervention and improving pipeline accuracy and reliability.
- Migrated legacy ETL workflows to Spark, improving performance and reducing infrastructure costs for large-scale batch processing jobs.
- Implemented security and compliance standards by configuring Kerberos authentication and HDFS permissions, ensuring compliance with corporate governance policies.
- Conducted Spark job tuning to optimize execution time by adjusting memory configurations, parallelism settings, and partition strategies for large-scale retail workloads.
- Built Python-based automation scripts for metadata management and data lineage tracking, improving governance and reducing operational overhead.
- Collaborated with cross-functional teams in Agile environments, ensuring timely delivery of data solutions aligned with business requirements.
- Worked on Git-based version control and Jenkins pipelines for code deployment and environment synchronization, ensuring smooth release management for all ETL workflows.
Education
Master of Science (MS) - Computer Science
Missouri State University
Skills
- Big data technologies: Hadoop, Spark, Hive
- Cloud computing expertise: AWS and Azure
- Python programming
- Database management: Oracle, Teradata, SQL Server, MySQL
- Data modeling expertise
- Proficient in Apache Airflow and Oozie
- Skilled in creating visual data reports
- Proficient in Git and GitHub
- Agile methodologies
Timeline
Data Engineer
TD Bank
08.2024 - Current
Data Engineer
Blue Cross and Blue Shield of Minnesota
12.2021 - 08.2023
Data Engineer
Home Depot
01.2020 - 12.2021
Master of Science (MS) - Computer Science
Missouri State University