Summary
Overview
Work History
Education
Skills
Websites
Certification
Timeline
Work Availability
Generic

Diya Nayak

Bellevue,WA

Summary

Overall 16+ years of experience in Information and Technology industry. Successfully completed the projects for MS Windows, Finance IT, AdCenter, Government, Healthcare and Corporate industries within the stipulated time. 7+ years of design, development and administration experience of multi-terabyte data mart and Data warehouse/ Business Intelligent system. Well versed in data modeling/reporting/validation, database architecture, ETL, dashboard development, software development life cycles (SDLC) and Big Data Technologies. 3+ years of data analysis experience enablement of database automation tools, data flow process reviews, Data Visualization, identification of data quality issues and related cleansing. Involved in the SQL DBA activities include but not limited to installation, configuration, importing/exporting, patch installs, capacity planning, backup and recovery of mission-critical databases. Proficient in interacting with users, analyzing client business processes, documenting business requirements, performing design analysis and developing design specifications. Experience with Azure Analytics stack, Azure Data Lake, Azure Databricks, Azure Data Factory. Experience of designing/building the OLAP databases, data modeling concepts (star schema, snowflake schemas, normalized & de-normalized data models), analyzing performance statistics and tuning. Expertise in developing Parameterized, Chart, Graph, Linked, Dashboard and Scorecards Report on SSAS/Tableau cubes using MDX and DAX scripting. Strong business modeling capabilities; experience in building complex, multi-sourced models in ways that enable easy/ automated quarterly and yearly roll-overs. Experienced in loading and analyzing large datasets with Hadoop framework (MapReduce, HDFS, PIG, HIVE, Sqoop, SPARK, Impala, Scala, Kafka), NoSQL databases like MongoDB, HBase, Cassandra. Worked in Agile/Scrum Methodology with daily stand up meetings and good knowledge in Team Foundation Server. Experience with SQL Azure Concepts and Windows Azure Platform. Hands-on experience solutioning and implementing analytical capabilities using the Azure Data Analytics platform including, Azure Data Factory, Azure Storage, Azure SQL Data Warehouse/Synapse, Azure Data Lake. Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills, excellent problem-solving abilities, and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed and utilizing excellent communication skills. Detail-oriented [Job Title] designs, develops and maintains highly scalable, secure and reliable data structures. Accustomed to working closely with system architects, software architects and design analysts to understand business or industry requirements to develop comprehensive data models. Proficient at developing database architectural strategies at the modeling, design and implementation stages.

Overview

17
17
years of professional experience
1
1
Certification

Work History

Data Engineer

Microsoft Corporation
8 2020 - Current
  • Working closely with clients and stakeholders to define software specification; Interfacing with business clients, gathering requirements and delivering complete data analytics and BI (Business Intelligence) solution
  • Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using Microsoft Azure technologies
  • Build data warehousing solution using Microsoft Azure technologies
  • Own the design, development, and maintenance of ongoing metrics, reports, analysis, dashboards, etc
  • To drive key business decisions
  • Recognize and adopt best practices in reporting and analysis: data integrity, security, privacy, test design, analysis, validation, and documentation
  • Continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for customers
  • Creating proof of concepts and prototypes to address business scenarios
  • Translate analysis and findings into visuals for non-technical (and technical) audiences and provide a clear view into data interpretation
  • Enable business to make clear tradeoffs between and among choices, with a reasonable view into likely outcomes
  • Maintain understanding of strategic goals, business challenges and customer needs
  • Write clean scripts for data analysis, ETL, and visualization
  • Prepare and present findings of investigations & solutions to stakeholder
  • Work/ lead talented team of engineers distributed across globe in offshore and onshore model
  • Proactively articulating status, issues, and resolution to team, leads, and project managers.

Data Engineer

Microsoft Corporation
10.2019 - 08.2021
  • Creating/maintaining scope scripts and jobs to generate daily stream file on COSMOS (Big Data Technology) of evidence
  • Designing, building, and maintaining large-scale multi-billion rows BI data models based on Azure Analysis Services / Tabular and PowerBI
  • Develop and maintain the dashboards, KPI's and scorecards using gauges and charts for analysis using Power Bi report
  • Analyze complex, high-volume (from COSMOS, SQL Server), multi-dimensional data from various sources to derive the useful metrics to be used for the marketing campaigns & executive dashboards
  • Participate in designing, building, maintaining, and improving WCB data platform, including Data Lake, SQL and non-SQL databases and data warehouses, data pipelines and BI systems
  • Internal and external customers developing PowerBI and Excel-based reports to leverage cubes as shared standardized and integrated data models
  • Develop solutions using Spark SQL, Spark streaming, Kafka to process web feeds and server logs
  • Design and Build Modern Data Pipelines and maintain data warehouse schematics, layouts, architectures and relational/non-relational databases for data access and Advanced Analytics
  • Partner with key stakeholders: senior business leaders, leaders across Field and Corp as well as other Finance stakeholders within Microsoft, contributing to making critical decision to drive business impact
  • Transferring and transforming data with Azure Synapse Analytics pipelines
  • Responsible in Working with various Python integrated development environments like PyCharm, Idle
  • Owning the maintenance, improvement and roll-over part of various models (e.g
  • COS, Productivity), PBI dashboards/ PBI desktop and various field portals by pulling data in real time: e.g
  • Mercury, MSS, KPI Lake, MS People and others
  • Develop views and templates with Python and Django & view controller and templating
  • Design and develop Azure stream analytics jobs to process real time data using Azure Event Hub and develop Batch application to read EventHub files and process that file to the ADLS
  • Build Scope script to update latest profile changes of Users in the final LTD which updates every hour to track user’s age group and parent information
  • Create wrapper job to backfill the email and the birthdates which is missed due to some technical glitches using ADF pipeline
  • Build data analytics solutions using Azure Synapse serverless SQL pools
  • Working with Data Warehouses using Azure Synapse Analytics.

Data Engineer

T-Mobile US Inc.
01.2019 - 10.2019
  • Gather business requirements to conceptualize, design and deliver new or ongoing production reports and visual dashboards
  • Design, develop and implement PowerBI Dashboards, Scorecards & KPI Reports using power query and Rest API
  • Setup and monitor SQL Server Database Security environments using profiles, database privileges and roles to prevent data theft and increase data integrity
  • Build Pipelines in ADF using linked Services/Datasets/Pipelines/ to extracts, transform and load data from different sources like Azure SQL, BLOB storage, Azure Data warehouse, write-back tool and backward
  • Create calculated measures and columns with DAX, visualizations, and complex calculations to manipulate the data as per business need in PowerBI desktop
  • Used Apache Spark and Azure Databricks to build the Data pipelines for different projects, as well as PySpark for Databricks Jobs
  • Export data from ADLS and create new Aggregation and Scoring table into Azure SQL database using Databricks
  • Export data from Log Analytics workspace to Azure storage account and run query data from Azure Data Explorer (ADX) using Kusto to perform detailed analysis based on metrics and platform logs
  • Used the Jupyter and IPython notebooks to execute the python modules, which generates the Fact data from database and other storage systems
  • Configured data pipelines using Azure Synapse Analytics and Data Bricks
  • Ingest data into ADX tables from ADLS storage accounts as well as copying Hive Table(files and folders) for a specific partition or complete data On-premise cluster to Azure Data Lake Storage, Copy ADLS folder based on a date partition to HDFS(v2)
  • Implemented multiple processes to improve team's consistency in delivering accurate, reliable, and timely data to Compliance teams such as data validation dashboards, email alerts, and DRI checklists
  • Define, build, schedule, manage, and monitor production workflows/jobs using Control-M to automate and orchestrate all phases in ETL/ELT
  • Built data pipelines with Azure synapse or Azure Data Factory using Python.

Data Engineer/Data Analyst

Quantum
01.2019 - 10.2019
  • Gather business requirements to conceptualize, design and deliver new or ongoing production reports and visual dashboards
  • Design, develop and implement PowerBI Dashboards, Scorecards & KPI Reports using power query and Rest API
  • Setup and monitor SQL Server Database Security environments using profiles, database privileges and roles to prevent data theft and increase data integrity
  • Build Pipelines in ADF using linked Services/Datasets/Pipelines/ to extracts, transform and load data from different sources like Azure SQL, BLOB storage, Azure Data warehouse, write-back tool and backward
  • Create calculated measures and columns with DAX, visualizations, and complex calculations to manipulate the data as per business need in PowerBI desktop
  • Used Apache Spark and Azure Databricks to build the Data pipelines for different projects, as well as PySpark for Databricks Jobs
  • Export data from ADLS and create new Aggregation and Scoring table into Azure SQL database using Databricks
  • Export data from Log Analytics workspace to Azure storage account and run query data from Azure Data Explorer (ADX) using Kusto to perform detailed analysis based on metrics and platform logs
  • Used the Jupyter and IPython notebooks to execute the python modules, which generates the Fact data from database and other storage systems
  • Configured data pipelines using Azure Synapse Analytics and Data Bricks
  • Ingest data into ADX tables from ADLS storage accounts as well as copying Hive Table(files and folders) for a specific partition or complete data On-premise cluster to Azure Data Lake Storage, Copy ADLS folder based on a date partition to HDFS(v2)
  • Implemented multiple processes to improve team's consistency in delivering accurate, reliable, and timely data to Compliance teams such as data validation dashboards, email alerts, and DRI checklists
  • Define, build, schedule, manage, and monitor production workflows/jobs using Control-M to automate and orchestrate all phases in ETL/ELT
  • Built data pipelines with Azure synapse or Azure Data Factory using Python.

Data Engineer

Windows Edge
01.2018 - 01.2019
  • Develop Cosmos (Azure Data Lake) streams with Scope (U-SQL) to prepare queries and create data pipelines that drive historical data analysis and enable insightful data visualizations with validated data engineering
  • Set data pipeline to pull data from Cosmos streams to Kusto (NOSQL Database)
  • Write efficient/complex queries in Kusto (a Big Data service for storing and querying), SQL to handle large amounts of unstructured/semi-structured data for processing
  • Developed Spark applications using Scala and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns
  • Data sources are extracted, transformed and loaded to generate CSV data files with Python programming and SQL queries
  • Automation of extraction, querying and storing resultant data process through C#, Kusto Client Library, SQL and PowerShell
  • Analyzed structured and unstructured data across disparate data source over 2TB data to drive business decisions: 10% operational efficiencies
  • Write queries in DAX, SQL, M, and Power query for SQL Server Analysis Services Tabular Data
  • Architect and implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB)
  • Write Scope Scripts to handle multi-million rows and for daily loading data from COSMOS Server and scheduling the data load using DataGrid
  • Write Kusto (KQL) and Cosmos Queries (SCOPE) and publish the data into Power BI.

Data Engineer

Windows Edge
01.2018 - 01.2019
  • Develop Cosmos (Azure Data Lake) streams with Scope (U-SQL) to prepare queries and create data pipelines that drive historical data analysis and enable insightful data visualizations with validated data engineering
  • Set data pipeline to pull data from Cosmos streams to Kusto (NOSQL Database)
  • Write efficient/complex queries in Kusto (a Big Data service for storing and querying), SQL to handle large amounts of unstructured/semi-structured data for processing
  • Developed Spark applications using Scala and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns
  • Data sources are extracted, transformed and loaded to generate CSV data files with Python programming and SQL queries
  • Automation of extraction, querying and storing resultant data process through C#, Kusto Client Library, SQL and PowerShell
  • Analyzed structured and unstructured data across disparate data source over 2TB data to drive business decisions: 10% operational efficiencies
  • Write queries in DAX, SQL, M, and Power query for SQL Server Analysis Services Tabular Data
  • Architect and implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB)
  • Write Scope Scripts to handle multi-million rows and for daily loading data from COSMOS Server and scheduling the data load using DataGrid
  • Write Kusto (KQL) and Cosmos Queries (SCOPE) and publish the data into Power BI.

BI Developer/Data Analyst

EPG and SMS&P Annuity Reporting
07.2011 - 09.2017
  • Key responsibilities included SQL Server migration, defining partitions on the tables, job definitions and scheduling, creating and managing user access via security groups
  • Creating, deploying and executing SSIS packages, creating and processing OLAP and Tabular cubes, defining SSRS and PowerBI reports, SharePoint integrations and Power Pivot modeling
  • Act as an SME to business and the technical aspects of the metrics collaborating with MS field and Account Managers worldwide to evaluate their concerns with respect to metrics and provide appropriate responses or decisions
  • Worked with other module leads in effort estimation for the releases, design discussion, code review and code quality analysis
  • Gained an overall 25% improvement in the monthly deployment process by proposing the performance improvement solution
  • With over 300 escalations a month, once an extremely concerned endeavor, helped strategize the escalation management in such a way that there has not been any notable concern for the last couple of quarters from the field on the process
  • Designed new reports, gathered requirements, analyzed data, developed and built SSRS reports and dashboard for executive view covering selected metrics and KPIs for sales excellence
  • Delivered training to simplify license management for global sales executive teams before the launch of OTRRR principles and scenario
  • Performed data analysis and data profiling using complex SQL, MDX and DAX on various sources systems including Azure SQL and Teradata
  • Created various PowerBI reports with rich visualizations that deliver Financial Insights based on Tabular cubes and Azure Data sources like SQL Azure, Azure Blob
  • Mentored Technical Support Engineers in researching, resolving and documenting customer server issues
  • Developed and implemented databases, data collection systems, data analytics and other strategies that optimize statistical efficiency and quality
  • Explored and proposed a design for automated override adjustment system reducing the manual intervention to 20%
  • Troubleshoot the data inaccuracies using COSMOS SCOPE scripts and automate the same.

SQL DBA/Service Engineer

ADCENTER
04.2010 - 06.2011
  • Key responsibilities included Taking care of client/server connectivity, query optimization, back-up/recovery, replications, clustering, log shipping and other DBA tasks
  • Managing multi-terabyte (>200TB) database environment and business intelligence systems
  • Drive root cause analysis and service improvements involving bug fixes in close partnership across several Tier 2/3 level engineering teams
  • Create and implement instance Health Check Monitor Alerts to Analyze Deadlocks and diagnose issues on the server
  • Managing and providing support to multi-terabyte cubes developed for Microsoft AdCenter business
  • Employed an automated process to monitor upstream and update domain data with 0% manual intervention
  • Built Job in pre-production environment that contains all the necessary steps to bring the system up and running in the least amount of time in case of any failures.

Programmer Analyst

JE MDK desktop – Global Analytics 2.0
03.2008 - 02.2009
  • Worked on designing of Star and Snowflake data models Schema used in relational, dimensional and multidimensional data modeling
  • Adept in defining referenced relationships and calculated members in SSAS and used MDX queries, creating OLAP structures from OLTP databases
  • Defined Query for generating Drill down reports, Matrix reports, Chart reports and handling parameterized reports, creating on-demand and scheduled reports in SSRS
  • Designed SSAS Tabular solutions improving the overall performance of the cubes by 25%.

SQL Developer

Maryland Pension Administration System
09.2007 - 02.2008
  • Created Scripts in T-SQL to construct triggers, tables, user functions, views, indexes, user profiles, relational database models, data dictionaries and data integrity
  • Generated some wrapper code in C# to get better maintainability
  • Created SSIS/DTS packages for Retiree’s Finance application that would transfer data among servers and scheduled the same SSIS packages by creating the corresponding job tasks.

Education

Bachelor’s in computer engineering -

Dharmsinh Desai Institute of Technology (DDIT)

Skills

C, Python, Java, VB, HTML, XML, ASPNET, MapReduce, Pig, Hive QL

Certification

Big Data Technologies, University of Washington (2018)

Timeline

Data Engineer

Microsoft Corporation
10.2019 - 08.2021

Data Engineer

T-Mobile US Inc.
01.2019 - 10.2019

Data Engineer/Data Analyst

Quantum
01.2019 - 10.2019

Data Engineer

Windows Edge
01.2018 - 01.2019

Data Engineer

Windows Edge
01.2018 - 01.2019

BI Developer/Data Analyst

EPG and SMS&P Annuity Reporting
07.2011 - 09.2017

SQL DBA/Service Engineer

ADCENTER
04.2010 - 06.2011

Programmer Analyst

JE MDK desktop – Global Analytics 2.0
03.2008 - 02.2009

SQL Developer

Maryland Pension Administration System
09.2007 - 02.2008

Data Engineer

Microsoft Corporation
8 2020 - Current

Bachelor’s in computer engineering -

Dharmsinh Desai Institute of Technology (DDIT)

Work Availability

monday
tuesday
wednesday
thursday
friday
saturday
sunday
morning
afternoon
evening
swipe to browse

Similar Profiles

CREATE PROFILE
Diya Nayak