Shanti Kopuri
Manchester,CT
Summary
Dynamic Lead Data Engineer with a 12+ years proven track record specializing in cloud migration and data pipeline optimization. Successfully enhanced ETL processes, databases and KPI's achieving a 30% increase in data processing speed. Adept at mentoring junior staff and fostering collaboration, ensuring data integrity and compliance across projects.
Overview
10
10
years of professional experience
Work History
Lead Data Engineer
Plymouth Rock Assurance
02.2024 - Current
- Designed, built and managed ELT data pipeline leveraging airflow and dbt models.
- Migrated Informatica workflows to DBT and Snowflake models/tables, contributing to data modernization efforts.
- Optimized Informatica jobs for performance improvements and provided production support for failure resolution.
- Collaborated with business analysts and data scientists to understand data requirements and transform them into effective ETL solutions.
- Developed Informatica mappings and scheduled jobs using Control – M scheduler to ensure timely data processing.
- Mentored and guided analysts on building purposeful analytics tables in dbt for cleaner schemas.
- Conducted analysis on current tables within DEV, UAT and PROD databases to remove the non-used tables as a part of POC to move to Snowflake cloud.
Lead Data Engineer
Eclinical Solutions
Mansfield, Massachusetts
10.2022 - 01.2024
- Led a team of 8 data engineers in the design and implementation of scalable and efficient data pipelines, resulting in a 30% increase in data processing speed.
- Developed and implemented best practices for data engineering processes, ensuring data quality, security, and compliance with industry standards.
- Collaborated with business analysts and data scientists to understand data requirements and transform them into effective ETL solutions.
- Conducted data profiling and cleansing activities to ensure data quality and integrity throughout the ETL process.
- Developed and maintained documentation for ETL processes, ensuring knowledge transfer and compliance with best practices.
- Participated in performance tuning and optimization of ETL jobs, resulting in 60% improvement in processing efficiency.
- Collaborated with data scientists and business stakeholders to understand data requirements and implement solutions that align with business goals.
- Conducted code reviews and mentored junior team members, fostering professional growth and adherence to coding standards.
- Designed and developed interactive and visually appealing reports and dashboards using Snowflake and reporting tools such as Tableau, Looker, or Power BI.
- Created and scheduled airflow jobs to load the DB2 database models to Amazon S3 buckets for further data integration and manipulations.
- Worked on CI/CD pipelines and SCD models as a part of ETL integration into SQL Databases.
- Implemented data visualization best practices, ensuring reports are accurate and user-friendly while aligning with business objectives.
- Worked on Mongo DB concepts such as locking, transactions, indexes, replication and schema designs.
- Deployed the tools Microsoft Azure Cloud Service (PaaS, IaaS), and Web Apps.
- Used SQL Azure extensively for database needs in Customer Lookup and migrated the Azure CXP Tools to HTTPS based authentication using SSL encryption.
- Improved application performance using Azure Search and T-SQL query optimization.
- Implemented Corp, 2FA and Role based authentication mechanism in Azure CXP Tools which uses Microsoft Azure Active Directory and DSTS (Datacenter Security Token Service).
- Implemented Merge feature for SQL Server Table smooth synchronization.
- Created SQL Server Integration Services (SSIS) packages for uploading different formats of files (Excel, Flat files, Access) into SQL Server.
- Set up SQL Server Linked Server to connect multiple Servers/databases.
- Implemented Azure Application Insights to store user activities and error logging.
- Generated and optimized already existing views to enhance performance for creating reports and SSAS cubes.
- Created and maintained Snowflake data models to support reporting requirements, optimizing for performance and efficiency.
Lead Data Engineer
Travelers Insurance
06.2019 - 09.2022
- Led the migration of on-premises data infrastructure to the cloud, resulting in reduction of infrastructure costs and improved scalability.
- Implemented data governance practices, including data cataloging and lineage tracking, to ensure data traceability and compliance with regulatory requirements.
- Developed and maintained complex MySQL queries and stored procedures, enabling timely access to critical business data.
- Experience in creating fact and dimensional model in MS SQL server and Snowflake databased utilizing Cloud base Matillion ETL tool.
- Participated in database design and normalization activities, ensuring data integrity and optimal performance.
- Collaborated with cross-functional teams to analyze and gather data requirements, translating them into effective SQL solutions.
- Designed and implemented ETL processes for large-scale data integration.
- Created and developed data load and scheduler processes for ETL jobs using Matillion ETL packages, optimizing data integration workflows.
- Utilized AWS Data Migration Services and Schema Conversion Tool with Matillion ETL tool to streamline data migration processes.
- Used Service Now tool as our post development process to move our code from Dev to UAT/PROD.
- Adjusted capacity using Amazon Auto-Scaling.
- Managed the integration of AWS services with on-premise resources, creating a hybrid cloud environment that increased flexibility and reduced operational costs by 20%.
- Involved in designing and implementing serverless architecture leveraging AWS Lambda and Amazon API Gateway.
- Developed and maintained data warehouse solutions, ensuring data consistency and availability for business intelligence and reporting.
- Collaborated with stakeholders to gather requirements, design data solutions, and provide technical expertise in data-related projects.
- Collaborated with data architects to design and maintain data models that met business needs and supported advanced analytics initiatives.
- Provided technical leadership in troubleshooting and resolving complex data processing issues, minimizing system downtime and improving overall reliability.
- Worked on an internal project with ETL integrations with Python Pyspark and developed airflow orchestration to create the files from CDR database and move data to AWS S3.
- Pulled data from multiple data sources and API sources like Oracle, Mainframe, Teradata, flat files, web services and applied required business functionality using ETL SSIS.
- Used SSRS reporting tool to create drill down and drill through reports for the business leads to view the public health data at the most granular level.
Sr SQL BI Developer/DBA
Up2Date Technlogies LLC
05.2018 - 05.2019
- Developed data model to meet business reporting requirements and support new tool development.
- Facilitated communication during Program Increment planning sessions and daily standups to clarify stakeholder requirements.
- Worked closely with Investigators and Legal, Business analysts and scrum master to evaluate and guide the selection of technologies required for the project.
- Carried out detailed Analysis on design and framework creation.
- Loaded Data into the SAP HANA Database using SLT, BODS and DXC.
- Brought data from ECC, MongoDB, Oracle and Flat files, also implemented ABAP Data Flows in BODS.
- Interfaced with SAP BW to bring in Data in SQL.
- Monitored database performance and identify opportunities for optimization.
- Tuned SQL queries and database configurations to enhance efficiency.
- SAP HANA Security activities like User Management, Assigning Roles to users and Managing Analytical Privileges to enable course gained and fine-grained access control.
- Developed SAS, SSIS, SQL jobs that interact with different platforms/systems like Mainframe, SQL Server, Oracle, DB2, Teradata, Hadoop etc.
- Created business reports for investigation and litigation utilizing SAS, Teradata Studio, Hue Editor, SQL, and Oracle queries.
- Used SAS analysis, tableau, excel pivots to analyze the data and create reports.
- Developed Microsoft Access Databases, including tables, queries, forms and reports, using standard IT processes, with data normalization and referential integrity.
- Used Python scripting to create SSIS packages.
- Leveraged advanced Excel skills, including pivot tables, lookups, and complex formulas, to enhance business process efficiency.
- Involved in migration of database and SSIS Packages and troubleshoot migration issues.
- Developed, deployed, and managed SSIS packages to perform various ETL operations from/to various sources and destinations using SSDT, SSIS, Import/Export Wizard for SQL Server.
SQL ETL/ Reports Developer
Up2Date Technlogies LLC
01.2016 - 04.2018
- Developed ETL coding to facilitate all the business requirements.
- Pulled data from multiple data sources like Oracle, Mainframe, Teradata, flat files, web services and applied required business functionality using ETL SSIS.
- Extensively used SQL for accessing and manipulating database systems.
- Collaborated with business analyst to analyze specifications and determine source data requirements for data warehouse migration, participating in design team and user requirement gathering meetings for transition from ThoughtSpot to Snowflake.
- Analyzed data to identify schemas for deletion and replication from ThoughtSpot to the new database.
- Supported migration tasks by creating validation plans that ensured data integrity throughout the process.
- Worked on end-to-end life cycle of project development (SDLC), right from the business requirement till the project deployment in production.
- Facilitated design discussions to develop appropriate solutions and identify necessary enhancements.
- Closely worked with the reporting team to ensure that correct data is presented in the reports pulling from snowflake.
- Documented the complete process flow to describe program development, logic, testing, and implementation, application integration, coding.
- Created Technical Design Document from the functional specification documents.
- Created and visualized dashboards using Tableau Desktop.
- Generated Tableau combination charts using Tableau visualization software.
- In-depth knowledge of Tableau Desktop, Tableau Reader, and Tableau Server.
- Administered user, user groups, and scheduled instances for reports in Tableau.
- Gathered business requirements from various sources like business docs, Functional teams, SMEs and report users.
Education
Masters - Information Technology
Southern New Hampshire University
New Hampshire01.2014
Bachelors - Computer Science
JNTUH
India01.2013
Skills
- ETL processes
- ETL development
- Data integration
- Data warehousing
- Data modeling
- SQL development
- SQL and databases
- Data pipeline optimization
- Cloud migration
- Data governance
- Business intelligence
- Advanced analytics
- Microsoft Azure
- Agile methodologies
- Mentoring junior staff
- Business understanding
Timeline
Lead Data Engineer
Plymouth Rock Assurance
02.2024 - Current
Lead Data Engineer
Eclinical Solutions
10.2022 - 01.2024
Lead Data Engineer
Travelers Insurance
06.2019 - 09.2022
Sr SQL BI Developer/DBA
Up2Date Technlogies LLC
05.2018 - 05.2019
SQL ETL/ Reports Developer
Up2Date Technlogies LLC
01.2016 - 04.2018
Masters - Information Technology
Southern New Hampshire University
Bachelors - Computer Science
JNTUH