Madhu M N
King Of Prussia,PA
Summary
Experienced Senior Data Engineer Lead with over 12+ years of experience in building Modern data solutions using Azure PaaS service, Databricks service to ingest, process, analyze, and visualize data for real-time insights and decision-making. Strong experience in Advancing Data warehouse like Lakehouse, Medallion architectures. Good understanding of Spark architecture with Databricks, Structured streaming and manage clusters in spark jobs. Excellent reputation on resolving Modern data problems, Big data processing, PySpark Optimization and improving customer satisfaction.
Overview
12
12
years of professional experience
Work History
Lead Data Engineer
IBM Corporation, United States
King Of Prussia, Penn
06.2022 - Current
- Design and implement ETL processes using Databricks jobs to process and transform raw data into a usable format for analysis and optimize Spark jobs for data processing, analysis, and machine learning tasks.
- Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Databricks workflow Jobs, Notebooks, Azure Data Factory, Logic App, Azure Functions, App Service and Service Bus. Data Ingestion to one or more Azure Services (Azure Databricks Lakehouse, Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks workspace, Azure Data Factory.
- Building Data Analytic Platform Utility on Azure Databricks cloud environment. Which follows medallion architecture to insert data into Databricsk Lakehouse, where Source data will be first ingested into Bronze layer, then will be standardized into Silver layer, finally will be transformed into Gold layer as per business requirements.
- Implementing an end-to-end streaming process in Azure Databrick. Ingest streaming data from sources like kafka, Writing PySpark or SQL code to perform data processing and transformations on the streaming data, Persist the processed data into Databricks Datawarehouse and visualize streaming data for real-time insights and decision-making using Power-BI
- Optimize PySpark code using some of Databricks defined optimizing techniques like Optimize, ZORDER, Partitioning, dataSkippingStatsColumns & some other functions.
- Good understanding on some of advanced datawarehousing architecture like Databricks Lakehouse and Medallion architecture. Storing persistent data in Delta Lake in form of Delta table.
Azure Data Engineer
IBM India Private Limited
Bangalore, Karnataka, India
08.2019 - 06.2022
- Analyze, design and build Modern data solutions using Azure PaaS service to support visualization Of data. Understand current Production state of application and determine the impact of new implementation on existing business processes.
- Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Databricks workflow Jobs, Notebooks, Azure Data Factory, Logic App, Azure Functions, App Service and Service Bus. Data Ingestion to one or more Azure Services (Azure Databricks Lakehouse, Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks workspace, Azure Data Factory.
- Implemented automated end to end solution for an EL & TL process using Data Factory. Logic App and Service Bus .
- Documentation of business/functional requirements & reviewed the solution architecture for ETL projects.
- Design auto ingestion of files into Azure storage, using Poly Base technology in order to access data stored in Azure blob.
- Involvement includes helping other team members resolve technical issues and analyze the complex business requirements. And also as Data Engineer Lead responsible for leading the development and optimization of our Databricks-based, Azure Data services based data processing and analytics platform.
Senior Data Engineer
Robert Bosch Engineering and Business Solutions Private
Bangalore, Karnataka, India
11.2018 - 07.2019
- Contributed Software engineering expertise in the development of application through the agile software development lifecycle, from designing requirement definition through successful testing and defect fixing.
- Responsible for writing stored procedures, functions and views, also used indexes efficiently for good performance of queries.
- Planned, co-ordinate analysis, design and extraction Of encounter data from multiple source systems into the data warehouse relational database while ensuring data integrity.
- Designed SSIS packages to transfer data from flat files. Excel SQL Server using Business Intelligence Development Studio.
- Extensively used SSIS transformations such as Lookup, Derived column, Data conversion, Aggregate, Conditional split, SQL task, Script task and Send Mail task.
- Performed data cleansing enrichment, mapping tasks and automated data validation processes to ensure meaningful and accurate data was reported efficiently.
ETL Developer
Genpact India Private Limited
Bangalore, Karnataka, India
11.2015 - 10.2018
- Obtained user approvals from the client for the collected requirements to ensure similar understanding between development team and business.
- Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and Controlling and granting database access and Migrating On premise databases to Azure Data lake store using Azure Data factory.
- Extensively utilized SSIS packages to create complete ETL process and load data into database which was to be used by Reporting Services.
- Identified the dimension, fact tables and designed the data warehouse using star schema.
- Encapsulated frequently executed SQL statements into stored procedures to reduce the query execution times.
- Created SSIS packages to implement error/failure handling with event handlers, row redirects, and loggings.
- Managed packages the in SSISDB catalog with environments; automated deployment and execution with SQL agent jobs.
SQL BI Developer
Tata Consultancy Services
Bangalore, Karnataka, India
12.2011 - 10.2015
- Involved in complete Software Development Life Cycle (SDLC) process by analyzing business requirements and understanding the functional workflow of information from source systems to destination systems.
- Designed and developed SSIS Packages to import and export data from MS Excel, SQL Server 2012 and Flat files.
- Involved in complete SSIS life cycle in creating SSIS packages, building, deploying and executing the packages in both the environments (Development and Production).
- Used various Transformations in SSIS Dataflow, Control Flow using for loop Containers and Fuzzy Lookups etc.
- Responsible to automate the process of loading raw data files to Data warehouse and integrating DW with Tableau server for auto refresh using power shell script and SSIS.
Education
Bachelor of Engineering - Telecommunication Engineering
Sri Siddhartha Institute of Technology
Tumkur, Karnataka, India05-2011
Diploma - Electronics And Communications Engineering
Government Polytechnic
Tumkur, Karnataka, India05-2008
Skills
- Spark Programming, Spark Dataframe API, Databricks Clusters
- Azure SQL Database, Azure Synapse Analytics
- Azure Data factory, Logic App, Azure App service, Azure Functions
- Service Bus, Azure Storage, Azure Dev-Ops services
- Azure Databricks, Databricks Workflow Jobs, Notebooks
- SQL, Python, PySpark, JSON
- MSBI (SSIS, SSAS, SSRS)
- MSQL Server programming, C# Programming
- Databricks Datawarehouse, Lakehouse, Delta Lake
Timeline
Lead Data Engineer
IBM Corporation, United States
06.2022 - Current
Azure Data Engineer
IBM India Private Limited
08.2019 - 06.2022
Senior Data Engineer
Robert Bosch Engineering and Business Solutions Private
11.2018 - 07.2019
ETL Developer
Genpact India Private Limited
11.2015 - 10.2018
SQL BI Developer
Tata Consultancy Services
12.2011 - 10.2015
Bachelor of Engineering - Telecommunication Engineering
Sri Siddhartha Institute of Technology
Diploma - Electronics And Communications Engineering
Government Polytechnic