PAVAN SRI HARSHA

· Primarily involved in Data Migration using SQL, SQL Azure, Azure Data Lake, and Azure Data Factory, GCP.

· As a support to architect, designed administrated and build analytic tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.

· Working with Source team to extract the data and it will be loaded in the ADLS Creating the linked service for source and target connectivity Based on the requirement.

· Once created, pipelines and datasets are triggered based on LOAD (History/Delta) operations.

· Based on source (big or small) data loaded files will be processed in Azure Databricks by applying operations in Spark SQL which will be deployed through Azure Data Factory pipelines.

· Involved in deploying the solutions to QA, DEV and PROD in azure Devops environment connecting through power shell to Azure.

· Professional in creating a data warehouse, design-related extraction, loading data functions, testing designs, data modeling, and ensure the smooth running of applications.

· Responsible for extracting the data from OLTP and OLAP using Azure Data factory and Databricks to Data Lake.

· Used Azure Databricks notebook to extract the data from Data Lake and load it into Azure, and On-prem SQL database.

· Worked with complete architecture and large data sets and high-capacity big data processing platform, SQL and Data Warehouse projects, Azure Synapse Analytics.

· Developed pipelines that can extract data from various sources and merge into single source datasets in Data Lake using Databricks.

· Generate and request certificates from trusted certificate authorities (CAs) or Azure Key Vault.

· Ensuring that the certificate's purpose (e.g., SSL/TLS, authentication) and key type (RSA, ECDSA) align with your data engineering needs.

· Store certificates securely to prevent unauthorized access. Azure Key Vault is a secure and centralized service for managing certificates and cryptographic keys.

· Deploy certificates to relevant Azure resources, such as virtual machines, Azure Kubernetes Service (AKS) clusters, or Azure App Service instances.

· Utilize Azure Automation or infrastructure-as-code tools (like Azure Resource Manager templates) for consistent and repeatable deployments.

· Utilize Azure RBAC (Role-Based Access Control) to grant the minimum required permissions to users and applications.

· Utilize Azure Private Link and Private Endpoints to ensure that communication between services and resources within Azure environment remains within the Azure network and is not exposed to the public internet.

· Implement multi-factor authentication for accessing the Azure Key Vault to enhance the security of certificate management.

• Based on source (big or small) data loaded files will be processed in Azure Databricks by applying operations in Spark SQL which will be deployed through Azure Data Factory pipelines
• Involved in deploying the solutions to QA, DEV and PROD in azure Devops environment connecting through power shell to azure
• Professional in creating a data warehouse, design-related extraction, loading data functions, testing designs, data modeling, and ensure the smooth running of applications
• Responsible for extracting the data from OLTP and OLAP using Azure Data factory and Databricks to Data lake
• Used Azure Databricks notebook to extract the data from Data lake and load it into Azure and On-prem SQL database
• Worked with complete architecture and large data sets and high capacity big data processing platform, SQL and Data Warehouse projects, Azure Synapse Analytics
• Developed pipelines that can extract data from various sources and merge into single source datasets in Data lake using Databricks.

• Developed PySpark pipelines which transforms the raw data from several formats to parquet files for consumption by downstream system
• Used Spark Streaming to receive real-time data from the Kafka and store the stream data to HDFS using Python
• Collected data using Spark Streaming from HDFS storage account in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS
• Built a POC on Spark jobs using python to load data from HDFS to Target DB
• Loading data from different sources to a data warehouse to perform some data aggregations for business Intelligence using python
• Design and build data structures on data warehouse and data processing using Pyspark with hortonworks platform to provide efficient reporting and analytics capability
• Bring structure to large quantities of data to make analysis possible, extract meaning from data
• Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2
• Used Sqoop to channel data from different sources of HDFS and RDBMS
• Developed Spark applications and Spark-SQL for data extraction, transformation and aggregation from multiple file formats
• Used SSIS to build automated multi-dimensional cubes.

• Develop database management systems for easy access, storage, and retrieval of data
• Perform DB activities such as indexing, performance tuning, and backup and restore
• Written new procedures by using Pl/SQL Collections and Exceptions
• Written new functions, and triggers as per business requirements
• Wrote Complex SQL Queries using joins, Subqueries, and analytical functions to retrieve data from the database
• Developed Data mapping, Transformation and Cleansing rules for the Data Management involving OLTP and OLAP
• Involved in creating UNIX shell Scripting
• Defragmentation of tables, partitioning, compressing and indexes for improved performance and efficiency
• Expertise in writing HadoopJobs for analyzing data using Hive QL (Queries), Pig Latin (Data flow language) and custom MapReduce programs
• Performed pig script which picks the data from one HDFS path and performs aggregation and loads into another path which later pulls populates into another domain.