P V Krishna

As a seasoned Data Engineer with over 7 years of experience, I have specialized in the design, implementation, and management of complex data pipelines and ETL processes, leveraging a comprehensive array of tools and technologies across Databricks, AWS, Azure, Snowflake, and the Hadoop ecosystem. Expertise in SQL, Python, and PySpark for designing and implementing data pipelines within the Databricks environment, optimizing data processing and analytics tasks. Proficient in developing Databricks Notebooks and applying transformations using Spark SQL for seamless data flow and integration across various platforms. Skilled in enhancing data integrity and availability through the construction and maintenance of data transformation processes, utilizing Scala and SQL within Databricks. Demonstrated expertise in working with Apache Spark and other distributed computing frameworks to process and analyze large-scale datasets, showcasing expertise in big data technologies. Implemented and automated data movement in and out of Snowflake, using features like Snowpipe, UDFs, and zero copy clones for efficient cloud data management. Created and managed Azure Data Factory (ADF) Pipelines, integrating Azure services like Azure Databricks and Azure Synapse Analytics for enhanced data processing capabilities. Configured and managed Azure Blob Storage and Azure Data Lake Storage, alongside AWS S3, for scalable data storage solutions. Developed and deployed machine learning models within Azure Machine Learning Studio and Databricks MLflow, leveraging AWS Lambda for serverless data processing tasks. Employed AWS Redshift for data warehousing, Amazon Athena for ad-hoc query executions, and AWS Glue for robust ETL processing, integrating tools like Apache Airflow for workflow orchestration. Advanced scripting and automation capabilities demonstrated through the development of Hive and Bash scripts, utilizing SQOOP and Hadoop Filesystem APIs for data ingestion. Hands-on experience in Hadoop administration and support, including the use of Cloudera Manager and Ambari for cluster management and maintenance. Utilized Java and Spring Boot to design and develop RESTful APIs within a microservices architecture, employing Docker and Kubernetes on OpenShift for containerization and orchestration. Implemented data streaming processes using Kafka, enhancing real-time data processing capabilities between microservices. Integrated Cassandra for NoSQL data storage and Elasticsearch for advanced search and analytics capabilities, showcasing versatility in handling different data storage formats like AVRO, Parquet, and ORC. Engaged in Agile Scrum methodologies, effectively employing tools like Confluence and Jira for project management and collaboration. Monitored application performance and managed cloud resources using Azure Monitor, Splunk, Kibana, and Elasticsearch for log analysis and performance metrics. Implemented security best practices in API development, including data encryption and authorization mechanisms, and integrated SonarQube and Fortify for continuous code quality and security assessments. Conducted code reviews and collaborated with QA teams to ensure the delivery of robust, error-free applications, highlighting a commitment to high-quality software delivery. Delivered CI/CD pipelines using GitLab, focusing on automation, continuous integration, and deployment to support a robust production environment. Committed to continuous learning and professional development, exploring emerging technologies like machine learning pipelines with TensorFlow and PyTorch, and blockchain for enhanced data security and transparency.