Over all 9+ years of professional IT experience and over 5+ years in Data Engineering, and 4+ years in Datawarehouse. Experienced data professional with a strong background in end-to-end management of ETL data pipelines, ensuring scalability and smooth operations. Proficient in optimizing query techniques and indexing strategies to enhance data fetching efficiency. Skilled in utilizing SQL queries, including DDL, DML, and various database objects, for data manipulation and retrieval. Expertise in integrating on-premises and cloud-based data sources using Azure Data Factory, applying transformations, and loading data into Snowflake. Strong knowledge of data warehousing techniques, including data cleansing, Slowly Changing Dimension handling, surrogate key assignment, and change data capture for Snowflake modeling. Experienced in designing and implementing scalable data ingestion pipelines using tools such as Apache Kafka, Apache Flume, and Apache Nifi. Proficient in developing and maintaining ETL/ELT workflows using technologies like Apache Spark, Apache Beam, or Apache Airflow for efficient data extraction, transformation, and loading processes. Skilled in implementing data quality checks and cleansing techniques to ensure data accuracy and integrity throughout the pipeline. Experienced in building and optimizing data models and schemas using technologies like Apache Hive, Apache HBase, or Snowflake for efficient data storage and retrieval for analytics and reporting. Strong proficiency in developing ELT/ETL pipelines using Python and Snowflake Snow SQL. Skilled in creating ETL transformations and validations using Spark-SQL/Spark Data Frames with Azure Databricks and Azure Data Factory. Collaborative team member, working closely with Azure Logic Apps administrators and DevOps engineers to monitor and resolve issues related to process automation and data processing pipelines. Experienced in optimizing code for Azure Functions to extract, transform, and load data from diverse sources. Strong experience in designing, building, and maintaining data integration programs within Hadoop and RDBMS environments. Proficient in implementing CI/CD frameworks for data pipelines using tools like Jenkins, ensuring efficient automation and deployment. Skilled in executing Hive scripts through Hive on Spark and SparkSQL to address various data processing needs. Collaborative team member, ensuring data integrity and stable data pipelines while collaborating on ETL tasks. Strong experience in utilizing Kafka, Spark Streaming, and Hive to process streaming data, developing robust data pipelines for ingestion, transformation, and analysis. Proficient in utilizing Spark Core and Spark SQL scripts using Scala to accelerate data processing capabilities. Experienced in utilizing JIRA for project reporting, task management, and ensuring efficient project execution within Agile methodologies. Actively participated in Agile ceremonies, including daily stand-ups and PI Planning, demonstrating effective project management skills.
Azure Administrator (Az-104)