Summary
Overview
Work History
Education
Skills
Certification
Timeline
Generic

VIJAYARAJ MANOHARAN

Minneapolis,MN

Summary

With over 17 years of exceptional development experience, including the last 8+ years specializing in Big Data and Hadoop technologies, as well as the MS Azure Cloud platform, I bring proficiency in data analysis, modeling, mining, landing, and machine learning techniques. My technical leadership has driven successful business outcomes, and I’ve continuously expanded my skills within the Data Engineering domain. Notably, I achieved technical certification from Databricks and played a pivotal role in upgrading and fine-tuning major pipelines. I also take pride in being a Databricks certified Spark Associate.

Overview

18
18
years of professional experience
1
1
Certification

Work History

Sr. Data Engineer

Target Corporation
10.2019 - Current
  • Engaged on development and implementation of end to end development of data ETL pipelines using Hadoop cluster and different big data analytic tools including Hive, Spark, Python, Scala, Sqoop, Kafka and Oozie
  • Involved in design and analysis of pipelines that imports data from internal target data sources to HDFS
  • Collaborated with cross-functional teams to define requirements and develop end-to-end solutions for complex data engineering projects.
  • Established standard procedures for version control, code review, deployment, and documentation to ensure consistency across the team''s work products.
  • Transforming according to business needs and publish the data in dashboards
  • Worked extensively in data transformations using scala, python and spark QL
  • Designed and implemented many pipelines for importing data and transformation of data following best coding standards
  • Implemented Automation test cases for unit testing using Scalatest library for code coverage. Created mock data for testing both positive and negative test cases.
  • Tuned performance in Hive using Partitions, Bucketing, Indexes and Parallelism concepts
  • Reengineered existing ETL workflows to improve performance by identifying bottlenecks and optimizing code accordingly.
  • Experienced in upgrading existing pipelines from Spark 2.x to 3.x versions along with Scala 2.11.x to 2.12.x
  • Created docker images of the pipelines to run in containerized environment to reduce runtimes
  • Implemented CDC (Change Data Capture) SCD (Slowly Changing Dimension) concepts in Hive tables
  • Migrated many hive job programs to Scala and spark for boosting performance
  • Solved performance issues in Hive and Scala with understanding of joins, groups, and aggregation
  • Experienced in data streaming using Kafka and Spark Streaming
  • Designed and Developed jobs that handles the Initial load and the Incremental load automatically using Oozie workflow
  • Implemented CICD model using GIT and Drone for seamless deployment
  • Adequate knowledge and working experience with Agile methodology
  • Sound working knowledge of displaying metrics in Grafana and alerting features
  • Contributed to internal activities for overall process improvements, efficiencies and innovation.
  • Worked on data quality checks and setup automatic alerts for every pipeline using oozie workflows and Amazon Deequ framework.

Sr. Data Engineer - Hadoop & Spark

3M Company
11.2015 - 10.2019
  • Engaged in pipeline building and maintaining in Hadoop cluster, handling substantial data volumes, and utilizing various data warehousing tools, including Hive, Spark, Python, Sqoop, Kafka, and Oozie.
  • Designed, Developed and maintained multiple PySpark data pipelines using Azure Data Factory, Azure Databricks and Synapse Analytics.
  • Collected requirements and created multiple Proof of Concepts (PoCs) in Azure cloud environment for a project proposal due to customer concerns about sluggish performance in ETL pipelines.
  • Ensured reliable data import and export between SQL Server and Azure Data Lake using Azure Data Factory.
  • Developed data transformations using HiveQL and SparkQL in Azure Databricks and Synapse.
  • Designed and implemented pipeline using Azure Databricks and written transformed data in Hive tables within HDInsights
  • Involved in creating map reduce jobs, UDFs and HIVE queries by following best coding standards
  • Implemented CDC (Change Data Capture) SCD (Slowly Changing Dimension) concepts in Hive
  • Worked on ETL Data Cleansing, Integration &Transformation using Hive and PySpark
  • Tuned performance in Hive using Partitions, Bucketing, Indexes and Parallelism concepts
  • Converted MapReduce jobs to PySpark for boosting performance
  • Solved various performance issues in Hive to optimize the pipelines by using compression codecs like Snappy.
  • Extensively worked on RDDs and dataframes in PySpark for processing data at a faster rate
  • Experienced in data streaming using Kafka and Spark Streaming
  • Lead the development of datalanding and transformation projects.
  • Imported data from critical applications to HDFS for data analysis and processing.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the R&D team
  • Designed and Developed jobs that handles the Initial load and the Incremental load automatically using Oozie workflow
  • Implemented CICD model using GIT and Jenkins for seamless deployment
  • Adequate knowledge and working experience with Agile methodology
  • Involved in unit testing activities and test data preparation for various business requirements
  • Moved between agile and waterfall approaches depending on project specifics and client goals, creating detailed project road maps, plans, schedules and work breakdown structures
  • Created and maintained technical documentation for launching Hadoop Clusters and for executing Hive queries
  • Worked on ETL reports using .Net reporting application and created statistics dashboards for business users
  • Involved in Scrum calls, Grooming and Demo meeting.

Senior Big Data Developer

Voya Financial
02.2014 - 10.2015
  • Data landing from different Data sources like (SQL server, CSV and text files) into HDFS using Sqoop and load into Hive tables
  • Developed Hive queries for analysis across different banners
  • Developed Hive UDFs in Python to import date into a structured format
  • Developed Oozie Workflows for daily incremental loads, which gets data from external applications and then imported into hive tables
  • Exported analyzed data from HDFS to RDBMS using Sqoop
  • Worked as a lead developer in projects involving SQL server and .Net technologies
  • Involved in development of new enhancements in MVC architecture using entity framework
  • Responsible to create new WCF web services using ASP.Net with C#
  • Used the AJAX tools like Update Panel etc for partial post back
  • Involved in Database diagram design and Tables design in SQL Server
  • Support/modify the exiting web services as per user requirements
  • Troubleshooting and bug fixing by identifying the root cause of problem
  • Developed and implemented SSIS packages for ETL jobs to import and transform data
  • Created and deployed SSIS packages to production, schedule them to run on routine basis and monitor error log for errors
  • Managing the SQL server databases and tuning performance
  • Provide Support to other teams to fetch the data from the centralized database using ASP.net XML Web Services
  • Manage the Dev, QA, Staging and Production environment
  • Worked in a collaborative development environment, providing assistance & technical guidance to fellow team members
  • Prepare technical design documents, perform code reviews, debugging and testing
  • Provide inputs in architecting, designing, and implementing a Service Oriented Architecture (SOA) for the application development group
  • Involved in Web Service testing using SoapUI tool
  • Created utility tools for estimation templates.

Lead .Net & SQL Developer

Chubb Insurance
02.2009 - 01.2014
  • Developed .NET library project using C# and provides middle layer components with HTTP Module and will be distributed to all users via MSI package
  • Created different business classes and interfaces using .Net Framework 4.5 with Object Oriented Programming Methodology
  • Designing and developing web forms using ASP.Net 4.5
  • Used CSS (Cascading Style Sheets) in ASP.Net pages
  • Created User Controls and Custom Controls for common purpose in the whole application
  • Used ASP.Net Telerik Kendo Controls for like grid view, autocomplete etc
  • Used XML for request and response from the WCF Services and manipulated the result as accordingly
  • Created the Custom User Grid View Control which includes the Sorting and Paging
  • Used Dataset, Data View and Data Adapter to manipulate and display data
  • Involved in relational database diagram design and Tables design in SQL Server
  • Created Stored Procedures, Views, Triggers and Complex T-SQL queries in SQL Server
  • Involved in Unit testing and Bug fixing
  • Involved in the Build and Release activities
  • Involve in creating the read only API’s application for other applications
  • Ingenuity around creation of reusable objects and engines
  • Facilitated problem solving and collaboration
  • Requirement analysis and understanding
  • Coordinated with offshore for day to day activities and assured quality deliverables to customer
  • Prepared high-level and detailed design documents
  • Designed and developed ETL jobs that handled the Initial load and the Incremental load.

.Net & SQL Developer

JP Morgan Chase
09.2006 - 01.2009
  • Involved in Analysis, Designing and coding and implementation of the application
  • Designed and developed web forms using ASP.Net 2.0
  • Design and develop front ends for application to support business rules using C#
  • Development of component classes and interfaces in C# encapsulating business logic with strict adherence to Object Oriented Paradigms
  • Used CSS (Cascading Style Sheets) in ASP.Net pages
  • Created own Controls library with Infragistics Controls
  • Created User Controls and Custom Controls for common purpose in whole application
  • Responsible for implementing and deployment of Web Services
  • Designed and developed complete front-end application using ASP.Net with C# as code behind for internal processing
  • Developed and hosting Web Services on IIS by following Service Oriented Architecture (SOA)
  • Implemented the functionality to render data using XML files and format them using XSLT
  • Involved in Database diagram design and Tables design in SQL Server
  • Created Stored Procedures, Views, Triggers and Complex T-SQL queries in SQL Server
  • Involved in development and execution of unit test cases
  • Created and deployed Windows Service for Emails
  • Implemented Email utility using windows service
  • Used Team Foundation Server (TFS) for project repository and versioning control
  • Prepared high level and low-level system design diagrams (class diagrams, sequence diagrams)
  • Effectively communicated with application representatives from both IT and Business to resolve issues.

Education

Bachelor of Engineering - Mechanical Engineering

Anna University
Chennai, India
06.2006

Skills

  • Big Data Ecosystems: Hadoop, MapReduce, HDFS, Druid, Hive, Sqoop, Oozie, Zookeeper, Kafka, Spark, Drone, Docker, Kubernetes, MS Azure

  • Programming Languages/Tools: Python, Scala, C#, VB Net, ASP Net

  • Operating System: Windows 7,8,10, Server 2012,2016, UNIX, Linux, and Ubuntu

  • Databases: RDBMS Oracle 9i/10g, MS SQL Server 2008 to 2016, MS Access, MySQL and HBase

  • Browser Language: HTML5, CSS3

  • Scripting Language: JavaScript, jQuery XML

  • Repositories: Git, TFS

Certification

  • Oracle Certified Associate, 2008
  • Databricks Certified Spark Associate, 2023

Timeline

Sr. Data Engineer

Target Corporation
10.2019 - Current

Sr. Data Engineer - Hadoop & Spark

3M Company
11.2015 - 10.2019

Senior Big Data Developer

Voya Financial
02.2014 - 10.2015

Lead .Net & SQL Developer

Chubb Insurance
02.2009 - 01.2014

.Net & SQL Developer

JP Morgan Chase
09.2006 - 01.2009

Bachelor of Engineering - Mechanical Engineering

Anna University

Similar Profiles

CREATE PROFILE
VIJAYARAJ MANOHARAN