Summary
Overview
Work History
Education
Skills
Certification
Timeline
Generic

Varun K

Plano,TX

Summary

Seasoned Data Architect adept at understanding mandates, developing plans, and implementing enterprise-wide solutions. Complex problem-solver with an innovative approach. Ready to bring several years of progressive experience and take on a challenging new role with growth potential.

Overview

5
5
years of professional experience
1
1
Certification

Work History

Data Engineer

Optum
, MN
02.2024 - Current
  • Architect and maintain scalable, real-time, and batch data pipelines using Azure Data Factory, Kafka, and Databricks. Implement ETL/ELT workflows to ingest, transform, and store data across ADLS, Snowflake, and SQL Server, ensuring reliability and scalability.
  • Develop and optimize Medallion Architecture (Bronze, Silver, Gold) using Databricks and Delta Tables. Leverage Kafka and Databricks Structured Streaming for real-time data processing, with low-latency transformations and analytics.
  • Ensure data governance, security, and compliance with Unity Catalog for access control and lineage tracking. Manage Azure Key Vault for credential storage, encryption, and secure access.
  • Optimize Snowflake data modeling, query performance, and storage. Fine-tune Spark jobs and Delta Lake performance in Databricks for efficient, large-scale data processing.
  • Automate deployments with CI/CD pipelines using GitHub Actions and Azure DevOps. Collaborate with cross-functional teams to design scalable data solutions, document processes, and support data-driven decision-making.

Data Engineer

Datapro Information Technology
Pune, Maharashta
08.2020 - 11.2022
  • Developed and managed ADF pipelines for ingesting data from Oracle and ADLS, implementing incremental loading, scheduling automation, and performance optimizations.
  • Designed and optimized PySpark/Scala scripts in Databricks for data cleansing, transformation, and performance tuning, improving processing efficiency and scalability.
  • Built and maintained fact and dimension tables in Azure Synapse, optimizing queries with indexing, partitioning, and materialized views for enhanced analytical performance.
  • Implemented Azure Key Vault and Managed Identities for secure credential storage and access control, while automating deployments through Azure DevOps CI/CD pipelines.
  • Monitored and troubleshot pipeline failures, job performance, and query workloads, ensuring data integrity, governance, and seamless collaboration with BI teams and stakeholders.

Education

Master of Science - Data Analytics

Indiana Wesleyan University
Marion, IN
08-2024

Skills

  • Programming languages: Python, SQL, Scala
  • Cloud Technologies: Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Key Vault
  • Big Data Tools: Apache Kafka, Apache Spark, Delta Lake
  • Databases and Warehouses: Snowflake, SQL Server
  • Packages and frameworks: PySpark, Delta tables
  • Productivity and DevOps: Git, GitHub, Azure DevOps, CI/CD Pipelines

Certification

  • Data Engineering on Microsoft Azure from Microsoft - DP 203
  • Databricks Accredited Lakehouse Fundamentals

Timeline

Data Engineer

Optum
02.2024 - Current

Data Engineer

Datapro Information Technology
08.2020 - 11.2022

Master of Science - Data Analytics

Indiana Wesleyan University

Similar Profiles

CREATE PROFILE
Varun K