Summary
Overview
Work History
Education
Skills
Certification
Timeline
Generic

Teja Reddy Lingampally

Irving,US

Summary

  • I have 3+ years of hands-on experience in Data Engineering and Analytics, specializing in building robust big data pipelines within the Hadoop ecosystem.
  • Expertise with tools such as Azure Data Bricks, Azure Data Factory, and Azure Data Lake.
  • In the realm of data engineering, I've utilized Scala with Spark and PySpark, creating ETL/ELT pipelines in Azure Data Factory (ADF) to seamlessly extract, transform, and load data from diverse sources including Azure SQL, Blob storage, and Azure SQL Data Warehouse.
  • I have executed data migration tasks into enterprise Azure cloud and Snowflake. Worked in Hive, allowing me to craft intricate queries and contribute to ETL architecture, data warehousing, and data integrations.
  • I have hands-on experience with real-time pipelines, particularly in managing structured streaming for fault-tolerant streams, checkpointing, offset management, and optimizing parallelism. Throughout my career, successfully navigated the complete project life cycle, covering design, development, testing, and implementation of both Client-Server and Web applications.
  • Good at Shell scripting and UNIX commands, and maintain proficiency in version control tools like GitHub and Bitbucket.

Overview

4
4
years of professional experience
1
1
Certification

Work History

Azure Data Engineer

Verizon
09.2022 - Current
  • Created Azure Data Factory Pipeline to load data from On-premises SQL Server to Azure Data Lake store
  • Extensively utilized Azure Data Factory for ingesting data from disparate source systems
  • Processed schema oriented and non-schema-oriented data using Scala and Spark
  • Utilize Azure’s ETL, Azure Data Factory (ADF) services to ingest data from legacy disparate data stores to Azure Data Lake Storage
  • Used Azure Synapse Analytics for Information Analysis
  • Designed SSIS Packages to extract data from various OLTP sources to MS SQL Server
  • Extensively worked with SQL Server Integration Services (SSIS) to design and create mappings using various transformations like, Conditional Split, Lookup, Aggregator, Multicast and Derived Column
  • Designed and Published Power BI Visualizations and Dashboards to various Business Teams for Business use and Decision making
  • Developed Talend Bigdata jobs to load large volume of data into S3 data lake and then into Snowflake
  • Working on DBT (ELT tool) connection setup with Snowflake and have used DBT cloud to execute ELT pipelines in Snowflake
  • Define virtual warehouse sizing for Snowflake for different types of workloads
  • Working with Jenkins and GitHub to implement pipelines
  • Used Devop’s to integrate Azure services and SQL databases.

Data Engineer

IBM India
05.2020 - 11.2021
  • Worked as Hadoop developer on a project which migrates existing code from Hadoop 2.0 to 3.x version (Scala, Spark, Hadoop, Hive, Java)
  • Worked on enhancements where spark jobs have to be upgraded to new version
  • Created Hive queries to retrieve the other applications data from central repository and incorporate them into batch process and performing application logic to come up with derived values based on rules
  • Worked on solving data quality issues identified by the business owners
  • Reviewed production execution of the process and optimizing the workflows by enabling parallel process for independent actions and tuning hive queries
  • Collaborated with developers and performance engineers to enhance supportability and identify performance bottlenecks
  • Followed Agile process in project as a team member by working on user stories/tasks assigned and updating the efforts and providing status regularly
  • Worked on PySpark code enhancements and end to end testing post migration
  • Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analysing & transforming the data to uncover insights into the customer usage patterns
  • Used data analysis techniques to validate business rules and identify low quality missing data in existing data.

Education

Master of Science - Data Science

Montclair State University
Montclair, NJ
12.2023

Skills

Programming Languages: Python, Scala,   SQL, SAS,   PHP, PL/SQL, NoSQL, Big Query, Cloud SQL

Big Data Tools: Apache Spark,   HBASE, HIVE, MAPREDUCE, Kafka, Airflow, HDFS

Database Technologies: HIVE, MySQL, SQL/PL-SQL, MS-SQL   Server, Oracle, Teradata

Cloud Services: Azure Data Factory, Azure Synapse Analytics

Version Control: Git, Bitbucket

ETL Tools: AWS Glue, Apache Nifi

Scripting: Bash, Shell

Visualization Tools: Tableau

Certification

Microsoft Certified Azure Data Engineer

Timeline

Azure Data Engineer

Verizon
09.2022 - Current

Data Engineer

IBM India
05.2020 - 11.2021

Master of Science - Data Science

Montclair State University

Similar Profiles

CREATE PROFILE
Teja Reddy Lingampally