Narendra Kumar Kotte
Dunwoody,GA
Summary
Results-driven Data Engineer with over 10 years of IT experience and more than 6 years focused on Azure Databricks and cloud-based data engineering solutions. Demonstrated success in designing and implementing scalable data pipelines and centralized data warehouses, while enhancing data accessibility using tools like Power BI, Snowflake, and Azure Data Factory. Proven expertise in advanced data transformations and troubleshooting complex systems, ensuring optimal performance under pressure. Aiming to leverage extensive experience to drive innovative data solutions in a dynamic environment.
Overview
11
11
years of professional experience
Work History
Senior Data Engineer
Muzzo Group
Atlanta, GA
05.2022 - Current
- Centralized data warehouse designed and implemented for Muzzo Group with Medallion Architecture.
- Leveraged Azure Databricks with Unity Catalog for data processing, ensuring secure, governed access to structured data across Azure Data Lake Storage Gen2.
- Facilitated seamless integration of CSV and JSON files into Azure Data Lake Storage Gen2 using Azure Data Factory.
- Processed and transformed raw data in the Bronze Layer using Azure Databricks notebooks, performing initial cleaning, filtering, and schema enforcement before moving to the Silver Layer.
- Applied business rules and advanced transformations in the Silver Layer to enrich data, preparing it for final consumption in the Gold Layer for reporting and analytics.
- Utilized Databricks Delta Lake features for ACID transactions, schema evolution, and efficient query performance across the Medallion Architecture layers.
- Created PySpark and Spark SQL scripts in Databricks to automate ETL workflows.
- Engineered data pipelines to load and transform data into Azure Synapse Analytics for structured storage and integration.
- Established materialized views in Azure Synapse Analytics to optimize query performance and enable faster data retrieval for business users.
- Orchestrated end-to-end workflows using Azure Data Factory, including triggers and pipeline executions, to manage data movement between Raw, Bronze, Silver, and Gold layers.
- Implemented CI/CD pipelines in Azure DevOps to streamline the development and deployment of data engineering solutions, ensuring version control with Git.
- Designed and deployed scalable data processing pipelines using Databricks and DBT for large-scale data transformation.
- Collaborated with cross-functional teams to implement data solutions using Azure Data Factory and ADLS, ensuring high availability and reliability.
- Integrated Databricks Unity Catalog to enforce role-based access control and data lineage tracking across ADLS Gen2 layers.
- Developed PowerShell scripts to automate metadata extraction and file movement tasks across storage layers.
- Transferred data pipelines from on-prem databases to Snowflake using SnowSQL for smooth ingestion.
- Designed and optimized data schemas in Snowflake using best practices for OLTP, OLAP, and dimensional modeling (e.g., Fact and Dimension tables).
- Utilized Snowflake's Time Travel and Cloning features for data auditing, versioning, and backup management.
- Skilled in loading and transforming data into Azure Synapse Analytics using various ETL (Extract, Transform, Load) tools and techniques, such as Azure Synapse Data Flow, Azure Data Factory, and third-party integration tools.
- Streamlined data accessibility and user experience by crafting views within Azure Synapse Analytics, simplifying data retrieval for reporting needs, and facilitating a more intuitive interface.
Senior Data Engineer
Sonic Automotive Inc
Atlanta, MI
11.2021 - 05.2025
- Served as primary technical and functional contact for On-Premise data source.
- Implemented Azure cloud services such as Resource Groups, Storage accounts, Data Factory, Databricks ensuring efficient data processing.
- Collected data from URL source using Azure Data Factory (ADF) and integrated into Azure SQL.
- Implemented ETL transformations utilizing Dataflow within Azure Data Factory (ADF), aligning with business requirements to streamline data processing workflows effectively.
- Developed and orchestrated Data Factory pipelines with diverse triggers, enabling automated and scheduled data movement and transformation tasks to meet organizational needs efficiently.
- Utilized Azure Key Vault to securely store keys and secrets.
- Configured email alerts via Logic Apps for ADF pipelines, ensuring timely monitoring of execution status.
- Designed and implemented databases, SQL tables, and views for reporting.
- Designed and implemented end-to-end data pipelines using Azure Data Factory and Python, ensuring efficient data processing and integration across various sources.
- Developed efficient CI/CD pipelines leveraging Azure DevOps.
- Provided leadership by conducting regular code reviews and mentoring junior team members, instilling best practices in Snowflake development to ensure code quality and team growth.
- Created views in Snowflake to enhance data accessibility and usability.
- Designed and implemented ETL jobs in Snowflake, incorporating transformations aligned with existing job designs to load Dimension, Fact, and Fact Summary tables, ensuring data integrity and consistency.
- Enhanced query efficiency and decreased analysis latency through implementation of materialized views in Snowflake.
- Enhanced database query flexibility and reusability by implementing parameterized stored procedures in Snowflake.
- Utilized Snowflake streams for incremental record processing, ensuring real-time data updates.
- Spearheaded Python-based development initiatives, orchestrating code that seamlessly reads and writes data to and from databases, showcasing versatility in data manipulation and integration.
- Directed proof-of-concept initiatives to validate real-time data transfer from source systems to Snowflake.
Data Engineer
Anthem
Marietta, GA
05.2020 - 11.2021
- Automated data quality checks and error handling processes to ensure the integrity and reliability of datasets.
- Used Apache Spark in Databricks as an ETL tool to process large-scale data transformation tasks. Developed Spark-Scala applications for data processing, ensuring high-performance data transformations across distributed systems.
- Created Shell scripts to manage Spark applications in Databricks.
- Leveraged Apache Spark as ETL tool for data processing tasks.
- Executed ETL jobs with transformations in Spark using Scala to populate Dimension, Fact, and Fact Summary tables in Databricks.
- Created ETL jobs with transformations based on current job designs in Spark using Scala.
- Monitored data pipelines, ETL jobs, and system performance in production using Databricks for real-time job execution monitoring.
- Assisted in the investigation and resolution of production issues, including debugging and fixing data inconsistencies, failed jobs, and performance bottlenecks in a timely manner.
- Worked closely with business teams, QA, and developers to identify areas for process optimization, data quality improvements, and implementing enhancements in the production environment.
- Utilized Spark DataFrames and Spark SQL in Databricks for data transformations and aggregations.
- Created a strategy for updating and inserting changes into existing Dimensions and Fact tables.
- Developed, built, and bundled dependencies for Spark applications using Scala IDE, Scala, and Maven.
- Developed lambda functions with Python for business logic implementation.
- Created incremental Spark jobs to extract data from source databases.
- Promoted code across multiple environments including development, pre-production, and master branches using Bitbucket.
- Led team coordination efforts to ensure sprint deadlines were met.
- Coordinated efforts with infrastructure, network, database, application and BI teams to maintain data quality.
Senior ETL Developer
Datavail
Hyderabad, India
08.2014 - 05.2020
- Directed the entire lifecycle of 4 concurrent ETL projects, ensuring timely completions.
- Designed and maintained robust database schemas supporting large-scale data operations.
- Created and fine-tuned advanced SQL queries using Joins, Subqueries, and Window Functions to efficiently manage data across different systems.
- Assigned tasks and delegated responsibilities to support teams, ensuring high-quality production support.
- Designed, developed, and maintained ETL workflows as SSIS developer.
- Served as lead SSIS developer, designing and developing complex ETL workflows.
- Executed seamless data migration using SSIS and SQL Server.
- Offered continuous production support, actively overseeing and resolving production job issues, reducing system downtime.
- Exhibited excellent SQL proficiency by crafting intricate queries, functions, and views while enhancing their performance.
- Enhanced ETL workflow performance by fine-tuning source qualifiers, transformations, and SQL queries.
- Crafted and executed data warehousing strategies, incorporating star schema modeling for seamless integration of multi-source data.
- Created Unix shell scripts for automation, file transfers, and data encryption/decryption.
- Coordinated with business analysts, developers, and stakeholders to align system design with business requirements.
- Led knowledge transfer sessions, documenting essential production processes and troubleshooting procedures.
- Engaged with clients like PF Chang's, GameStop, and Activision to meet customer needs.
Education
Bachelor of Arts - Electronics And Communication Engineering
JNTU- Anantapur University
India Andhra Pradesh06-2012
Skills
- Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Snowflake, Delta Lake, Spark
- Python (Pandas, PySpark), SQL, Scala
- SSIS, Azure Data Factory, DBT
- SQL Server, MySQL, PostgreSQL, Snowflake
- Git, Jenkins, JIRA, Datadog, DPA
- Power BI
Timeline
Senior Data Engineer
Muzzo Group
05.2022 - Current
Senior Data Engineer
Sonic Automotive Inc
11.2021 - 05.2025
Data Engineer
Anthem
05.2020 - 11.2021
Senior ETL Developer
Datavail
08.2014 - 05.2020
Bachelor of Arts - Electronics And Communication Engineering
JNTU- Anantapur University
Similar Profiles
Drew DicksonDrew Dickson
Project Manager at Flournoy Construction Group, A Kajima USA GroupProject Manager at Flournoy Construction Group, A Kajima USA Group
Emmanuel G. FreemanEmmanuel G. Freeman
Project Manager at Group Fitness Challenge, 325th Medical GroupProject Manager at Group Fitness Challenge, 325th Medical Group
Sharon Jayne Butler-BarnesSharon Jayne Butler-Barnes
Customer Service Representative at Citrus Group / Woolworths GroupCustomer Service Representative at Citrus Group / Woolworths Group