Summary
Overview
Work History
Education
Skills
Education Certifications
Personal Information
Timeline
Generic

Senthilkumar Dhinakaran

Shakopee,MN

Summary

  • With over 18 years of extensive experience in working with advanced data analytics and data engineering, Agile project & Program/Product management, MLOps/DevOps using ETL, Big Data and Cloud Technologies.
  • Proven leadership abilities with a successful track record of leading multiple teams in both India and the USA
  • Specialized expertise in Marketing, Merchandising, Supply Chain, and Own-Brands domains
  • Recognized as a hands-on and proactive troubleshooter, who can rapidly identify business problems, formulate strategic approaches, initiating change, and implement new process in challenging and diverse environments
  • Self-motivated, organized and passionate team player with robust problem solving and analytical skills and who is totally committed to the organization’s goals.

Overview

19
19
years of professional experience

Work History

Lead Data Engineer

Guest And Marketing Data, Target
11.2019 - Current
  • Revolutionized the availability of Target Marketing data, including Roundel, PbR, RMS and Campaign data, for seamless business user consumption, enabling advanced analytics and reporting capabilities.
  • Developed a unified framework to migrate from hive to Spark Scala 2.3 and adhering to best ETL practices for the data engineering organization.
  • Communicated new or updated data requirements to product/business teams.
  • Engineered data caches and API interfaces for core datasets, boosting query accessibility efficiency by 30% through the decoupling of platform resource limitations.
  • Successfully migrated over 100+ core certified marketing datasets and 3 ML feature datasets with more than 500 features from legacy solutions to modern CICD Spark pipelines.
  • Optimized data processing by implementing cutting-edge data engineering techniques and tools.
  • Effectively refactored code base and improved job performance for the entire project,increasing runtime efficiency 20 to 30% for problematic pipelines by leveraging a profound understanding of parallel processing, partitioning, bucketing, sorting, and repartitioning. Additionally, implemented optimal memory and core configurations in Spark/Hive.
  • Developed unified Framework for Spark 3 migration and publishing as an artifact, supporting both Spark 2.3 with Scala 2.11 and Spark 3.4.1 with Scala 2.12.
  • This strategic framework implementations not only facilitated a smooth migration to Spark 3.4.1 but also resulted in a remarkable average 35% improvement in runtime efficiency for all spark jobs
  • Developed unified Framework for Hadoop 2 decommission - automated tracking of legacy decommissions data assets to keep track of the dataset freshness, file count, table/DB size and was able to see the impact after decommission process.
  • Cleared 1820 datasets gaining 122 TB of space for my team and overall ~120K datasets gaining 20 PB of space.
  • Unified CI/CD Framework Architect - Designed, developed, and implemented a robust CI/CD framework using Vela to mitigate risks, ensuring streamlined, improvement of project timelines (slashing over 4 hours/pipeline) and seamless automated deployment processes for data engineering
  • Unified Framework for Enterprise Pipeline Observability metrics - Regularly monitors more than 1,000 NUAs, supervising the status of 5,450 Oozie jobs, and approximately 170,000 YARN applications every 10 minutes to identify and optimize long-running jobs, resulting in a 20% reduction in support overhead
  • Enterprise Hackathon-Participation - Leveraged GenAI’s advanced NLP capabilities to implement following use cases 1) Generate user prompts and provide forecasted impressions for specified campaigns. 2)Analyze product attributes and desired themes, creating compelling and contextually relevant campaign content.

Senior Data Engineer

Vendor Report Card, Target
01.2017 - 11.2019
  • Merch Vendor Team builds & manages all data sources from modern / new sources (Kafka / API) which are related to supply chain to measure vendor performance and process compliance based on their OTS and FR percentages.
  • Built Spark Streaming pipeline to consumer messages Kafka for every 30 seconds and store them in Postgres and ingest to Green Field (Druid) reporting
  • Lead the team in building the Next Gen vendor compliance data by creating data pipeline in Azure Data Factory and Databricks using Notebooks in Spark/Python.
  • Created DAGs in airflow (Python) using the different operators (bash, python, hive, SSH, etc.) to orchestrate and schedule the data loads.
  • Build Grafana dashboard by ingesting data into influxDB and create alerts (Slack & Service-now) during abnormality
  • Delivered exceptional results under tight deadlines, consistently prioritizing tasks effectively to meet project timelines without compromising quality or accuracy.
  • Implemented hybrid solution to keep on-premises data stores up to date

Senior Hadoop Developer / Project Manager

Vendor - Reporting & Bulk Data Processing, Target (TCS)
12.2010 - 12.2016
  • Migrate the following Applications to Hadoop - Vendor Data Warehouse (VDW), Enterprise Data Warehouse (EDW), Guest Contact Analysis (GCA)
  • Manage multiple teams across India and USA
  • Work with business users to understand their stability problems & provide solutions
  • Develop strategic road map for each product based on the business requirements
  • Arrive at high-level solution that aligns with strategic roadmap and program objectives
  • Instrumental in providing story point estimates & contribute towards sprint planning
  • Work on high level architectural design and review team’s technical deliverables
  • Participate in Sprint planning meeting and daily stand-up meeting to discuss the status of assigned user stories
  • Gather new requirements from business users and incorporate during migration
  • Sqoop the data from DB2 & Oracle database to HDFS
  • Designed Hadoop pipelines for DataStage flows & Develop pipeline using HiveQL & pig scripts
  • Created workflows using Oozie xml and scheduled them from Control M & used Kerberos authentication from edge node to Hadoop clusters
  • Used HBase to keep the application common parameters
  • Work with business users for User Acceptance testing (UAT)
  • Created proofs of concept for innovative new solutions.
  • Tested troubleshooting methods and documented resolutions for inclusion in knowledge base for support team use.

Sr Application Developer

Enterprise Billing System, Pershing LLC
07.2005 - 11.2010
  • Comprehensive, flexible billing platform to provide billing for all trade and non-trade transactions executed by various IBDs contracted with Pershing
  • Prepare System Requirement Design (SRD),
  • Prepare System External Specification (SES), and System Internal Specification (SIS)
  • Developed reusable components which reduced 30% of manual effort.
  • Develop the Test cases Test scripts & Test plan
  • Conduct impact analysis of interface applications
  • Create low and high-level design documents
  • Deploy Cobol & JCL changes via NED

Education

Bachelor of Science - Mechatronics

Anna University
Chennai, India
05.2005

Skills

    Languages: Python, Scala, Hive Query Language (HQL), SQL,Unix Shell Script, Java

    Workflow Orchestration: Airflow, Oozie, Control-M

    Big Data Technologies: Hadoop Framework and its ecosystems (Sqoop, Hive, YARN), Spark, Kafka,NoSQL database (Cassandra, MongoDB), In Memory Database (SAP Hana, Redis, Druid, Influx), Relational Database (Teradata, MySQL,DB2, PostgreSQL)

    DevOps: Agile, GitHub, Jenkins, Drone, Vela, Gradle, SBT, Docker, Maven

    Reporting Tools: Domo, Greenfield, Grafana for observability metrics, Tableau, Microstrategy

    Cloud Technologies: Microsoft Azure (Data Factory, Azure Dataricks, Synapse, Azure Data Lake storage, Cosmos DB, data bricks), AWS (S3, EMR), GCP (Google Cloud Storage, Object Storage, Data proc)

    API Tech Stack: Kotlin, Spring boot, Swagger, Gradle, Docker, TAP (Target Application Platform), DB2, PostgreSQL) GCP, Kubernetes

    AI: Azure OpenAI, Vertex AI, LangChain, PaLM2, Chroma

    Test framework: Scala test, SonarQube

    ETL Tool: DataStage 85 & Informatica

    Defect Tracking: HP Quality Center, ServiceNow and JIRA

    CI & CD: GIT, Jenkins, Docker, Artifactory, Drone

Education Certifications

  • Microsoft Azure Data Engineering associate
  • Cloudera & MapR trained and certified associate (CCA-175)
  • IBM Spark Fundamentals Certified
  • Agile Product Ownership Certification (ICP-APO) certification
  • Scrum Master Accredited certified
  • Google Cloud Platform Fundamentals: Big Data & Machine Learning
  • Bachelor of Engineering in Mechatronics, Anna University, India

Personal Information

Marital Status: Married

Work Authorization: GC-EAD

Timeline

Lead Data Engineer

Guest And Marketing Data, Target
11.2019 - Current

Senior Data Engineer

Vendor Report Card, Target
01.2017 - 11.2019

Senior Hadoop Developer / Project Manager

Vendor - Reporting & Bulk Data Processing, Target (TCS)
12.2010 - 12.2016

Sr Application Developer

Enterprise Billing System, Pershing LLC
07.2005 - 11.2010

Bachelor of Science - Mechatronics

Anna University

Similar Profiles

CREATE PROFILE
Senthilkumar Dhinakaran