Summary
Overview
Work History
Skills
Timeline
Generic
Divya Kadiyam

Divya Kadiyam

Milford,MA

Summary

Data Engineer with 8+ years of experience in Building Data intensive applications, tackling complex architectural and scalability problems. Excellent analytical, problem solving, decision making and presentation skills. Can adapt to any newer technology or software at a greater pace.

Overview

11
11
years of professional experience

Work History

Biz Intelligence Database Administrator

Ericsson
08.2023 - 06.2025
  • Administered Snowflake environments for over 500 users by provisioning users, roles, and managing permissions to enforce enterprise RBAC policies, improving access control efficiency by 40%.
  • Configured Power BI Single Sign-On (SSO) with Snowflake using Azure AD OAuth, reducing user login time by 60% and supporting 10+ BI teams.
  • Managed dynamic role provisioning via Azure AD dynamic groups, reducing manual intervention and support tickets by 70%.
  • Designed and implemented row-level security and dynamic column masking policies protecting 100% of PII/PHI data across 12+ schemas, ensuring SOX/GDPR compliance.
  • Configured and managed Snowflake Data Shares across 6+ accounts, enabling secure, cross-org data access while cutting duplicate storage by 30%.
  • Deployed replication and failover processes for critical datasets, achieving
  • Automated user provisioning with SCIM/SAML via Azure AD, improving onboarding SLAs by 50%.
  • Built and maintained 25+ Terraform modules for Snowflake access controls and infrastructure, ensuring 100% IaC compliance and zero drift.
  • Monitored compute usage via Resource Monitors, optimizing warehouse configurations to reduce monthly cost by 20%.
  • Audited platform activity using Query History and Account Usage views, identifying and resolving performance bottlenecks, improving query efficiency by 25%.
  • Managed security configurations including network/session policies, key pair authentication, and OAuth, enhancing platform security posture by 2 compliance tiers.
  • Implemented Snowpipe and COPY pipelines handling 1M+ records/day, increasing data freshness and reliability.
  • Led quarterly security/compliance audits, resulting in zero audit findings and sustaining 100% compliance with regulatory standards.
  • Coordinated Snowflake upgrades quarterly and adopted new features based on release notes, increasing feature adoption by 35%.
  • Partnered with security and data teams to classify 100% of sensitive assets using Atlan, cutting audit prep time by 30%.

Senior Data Engineer

Indeed
12.2021 - 03.2023
  • Snowflake team engineer that supports the entire Indeed organization with 10000+ employees.
  • Tech Stack - Snowflake, Terraform, RBAC, RCLS, Data Masking
  • Designed and implemented secured access to snowflake using techniques like password rotation, SSO, Key pair, Oauth.
  • Tech Stack - Lambda, AWS Secrets Manger, Snowflake, Terraform
  • Snowflake automation of access control, data management, resource allocation.
  • Data Integration with Spark, DBT, AWS Glue, S3.
  • Built Executive dashboards in Tableau using Prep Builder on data sources like Snowflake, Tableau, Google Big Query and created unified organizational reports.

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021
  • Kung fu factory (Client) Data from external sources need to be processed at near real-time cadence using AWS services like Kinesis, Glue and Load the data to Redshift:
  • Created Kinesis Stream and integrated with data ingestion service to capture telemetry data in real-time and load into s3.
  • Created Glue ETL jobs to process the Json data from S3 and Copy the data to redshift.
  • Worked with Client to derive the schemas for the events and applied dynamic mappings of the columns and datatype to process the data efficiently.
  • Did some tuning and optimizations in the code using Spark to faster data processing.
  • Converted the data in parquet format and stored the data in S3.
  • Added Glue ETL triggers effectively to process the data on multiple parallel threads.
  • Created a simple Python callback function using Fastapi to process the callback and transform to Json and send it to firehose.
  • Imported redshift tables and view onto Tableau. Built dashboard in tableau to help game developers.
  • Tech Stack - Python, S3, AWS Glue, Redshift, Jupyter notebook, Pyspark, FastApi, EC2, Firehose, Postgres, Json, Parquet, Tableau.
  • Built a metric/alert driven actions using AWS lambda:
  • Monitor the CloudWatch metrics and create conditional action for high CPU or disk usage.
  • Based on the condition flag trigger a lambda function to take an action to increase the disk space or add more nodes.
  • Same concept is applied to custom jobs in Jenkins, where a lambda trigger a Jenkins webhook to run a job for a specific alert on the system.
  • Tech Stack - Cloud Watch, Lambda, Jenkins, Webhook, Event bridge.

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021
  • Branch Energy (Client) Raw third-party data need to be flattened and loading into snowflake effectively.
  • Created Airflow ETL jobs to extract third party data from different Rest endpoints.
  • Created custom copy command job to load bulk data from s3 to Snowflake via Airflow.
  • For continuous loading of small files from s3 setup a snow pipe configuration to auto load data to Snowflake.
  • Create Snowflake accounts and managed permissions for various users.
  • Setup adhoc data warehouses in snowflake for better administration.
  • Built some basic data models in snowflake via Airflow orchestration.

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021
  • Safe Graph (Client) Safe Graph has large geographical datasets the are taking hours to days to process the data.
  • The datasets needed to be processed soon and build some plots and graphs for few use cases from the data.
  • Used Pyspark to join the large Geographical datasets and for data cleanup.
  • Used Broadcast join, UDF functions for some datasets transformations.
  • Worked on Jupyter Notebook environment efficiently.
  • Compared Pyspark and spark koalas for the usecase.
  • Implemented a rolling window function with pyspark.
  • Derived meaning plots and graphs based of some mathematical calculations averaged by state.

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021
  • Nesterra (Client) Build a ETL Automation Framework to extract daily reports emailed from customer.
  • Apply transformation and load the data onto Azure Sql Databases.
  • Send automated notification of the ETL process.
  • Requirement is to extract excel/csv email attachments, validate, cleanup, process and load the data to Database.
  • Later create views on the tables and some store procedures based on the use cases.
  • Implemented a custom email scraper in Python to check the emails of the customer at regular intervals and extract the csv attachments onto cloud storage (Like Aws S3).
  • Worked on complex transformation within the ETL using python.
  • Did schema validation using third-party tools and cleaned up corrupted data.
  • Generated a Postgres connection from Python and ran complex Sql queries to load the data to Sql database.
  • Created an audit registry of the ETL workflow to maintain idempotency of the ETL.
  • Used Airflow as an Orchestration tool for the workflow which is hosted on AWS EC2.
  • Implemented a Custom Alerting tool that sends custom email alerts to the customers for any ETL failures or system outages.

Data Analyst

Gallagher operations & support services
08.2014 - 07.2017
  • Produced monthly reports using advanced Excel spreadsheet functions.
  • Created various Excel documents to assist with pulling metrics data and presenting information to stakeholders for concise explanations of best placement for needed resources.
  • Utilized data visualization tools to effectively communicate business insights.
  • Generated ad-hoc reports to evaluate specific business requirements.

Skills

  • Python
  • SQL
  • AWS - Glue
  • Snowflake - Snow Pipe
  • Snowflake - Data Masking
  • Snowflake - RBAC
  • Snowflake - RCLS
  • Azure
  • Tableau
  • Airflow
  • Kafka
  • Atlan

Timeline

Biz Intelligence Database Administrator

Ericsson
08.2023 - 06.2025

Senior Data Engineer

Indeed
12.2021 - 03.2023

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021

Senior Data Engineer/Analyst

Upwork
11.2018 - 11.2021

Data Analyst

Gallagher operations & support services
08.2014 - 07.2017

Similar Profiles

CREATE PROFILE