Ahemad Ali Shaik
Sr Data Engineer/Big Data/Cloud Migration
Irving,TX
Summary
Detail-oriented Senior Data Engineer designs, develops and maintains highly scalable, secure and reliable data structures. Accustomed to working closely with system architects, software architects and design analysts to understand business or industry requirements to develop comprehensive data models. Proficient at developing database architectural strategies at the modeling, design and implementation stages.
Overview
12
12
years of professional experience
4
4
Certifications
Work History
Lead Data Engineer, Client- Verizon
Prodapt Solutions
08.2023 - Current
- Streamlined data ingestion processes to accommodate increasing volumes of incoming information while maintaining data integrity and ensuring timely accessibility across organization.
- Delivered high-quality code reviews for colleagues'' contributions following established coding standards and best practices, maintaining consistency throughout projects.
- Developed scalable infrastructure capable of handling vast amounts of structured and unstructured data, improving overall system performance.
- Aligned business objectives with technical requirements through close collaboration with product owners providing insights based on available data sources.
- Optimized Kubernetes cluster recourse allocation to reduce costs.
- Reviewed Druid cluster configuration to improve overall performance of application in terms of data ingestion and Data retrieval.
Lead Data Engineer, Client-Verizon
Prodapt Solutions
10.2022 - 07.2023
- Worked on spark SQLs to BigQuery SQL conversion
- Optimize SQLs and modify table schema to improve Query performance and reduce Query cost
- Historical data load from Apache Hive to BigQuery tables
- Build Airflow DAGs to orchestrate and schedule pipelines based on existing Oozie jobs
- Validate data quality by comparing results of data pipelines before and after modernization
- Work with UAT team to get sign-off for modernized data pipelines.
- Generated detailed studies on potential third-party data handling solutions, verifying compliance with internal needs and stakeholder requirements.
- Solution decreased job failures and improved query performance by 30% and overall cost $10k per day
Technical Lead, Client-Verizon
Prodapt Solutions
04.2022 - 09.2022
- Design solution to move and modernize on-premises data lake to Google cloud
- Evaluate different services in GCP with MVP to estimate overall migration cost
- Design migration of data pipelines from on-premises to GCP DataProc/Composer DAGs
- Design architecture to implement observability in Airflow data pipelines
- Design data pipelines to move from spark-based framework pipelines to GCP BigQuery/Airflow DAGs
- Migrated on premises based Spark jobs to cloud with improved performance, reducing costs and enhancing efficiency of computing tasks.
- Modernized solution saved $10k per day for customer.
Data Engineer,Client-Windstream
Prodapt Solutions
06.2021 - 03.2022
- Worked on building pipelines for automating data retrieval from Oracle to HDFS
- Developed scripts using spark Dataframes to perform complex data joins
- Designed Cassandra data model for low latency and high availability
- Worked on performance tuning of Cassandra DB and spark applications
- Airflow DAG for automation and data refresh on daily basis
- Developed APIs to fetch data based on Business needs.
- Solution automated data pipelines and reduced system downtime.
- Application was able to achieve all KPIs planned and improved efficiency by 20%
- Provided technical guidance and mentorship to junior team members, fostering a collaborative learning environment within the organization.
- .Fine-tuned query performance and optimized database structures for faster, more accurate data retrieval and reporting.
- Increased efficiency of data-driven decision making by creating user-friendly dashboards that enable quick access to key metrics.
- Implemented real-time data processing solutions for timely decision-making and improved customer experience.
Graph Database Developer,Client-Windstream
Prodapt Solutions
08.2020 - 05.2021
- Involved in designing application architecture and building data model
- Worked with various end users to decide graph properties like vertices and edges
- Built and maintain data pipelines to pull data from Oracle
- Developed code using Spark and Cassandra to load data into Graph with relations
- Created Elasticsearch indexes for backend data to improve performance
- Developed APIs on top of graph databases.
- Solution reduced time taken by networks planning team by 70%
- Enhanced data integrity through the design and enforcement of referential integrity rules and constraints
- Reviewed peer-developed codes using dedicated version control systems ensuring adherence to coding guidelines and optimization principles
- Developed custom tools to automate routine tasks, increasing overall productivity for the development team
- Provided training sessions for junior developers on best practices in SQL programming techniques aimed at improving code efficiency
Bigdata DevOps Engineer,Client-Windstream
Prodapt Solutions
01.2017 - 07.2020
- Worked on Design, Develop, Monitoring, Tuning and Optimizing, Governing Large Scale Hadoop Cluster
- Monitored and Analyzed Job Performance, File system/Disk space Management, Cluster & Databases connectivity
- Created Azure CI/CD pipelines for all environments (Dev
- Test and Prod) for automated code deployment
- Configured and maintained Cassandra, Elasticsearch, Mongo and Airflow high availability clusters
- Built Kafka cluster and integrated with NIFI for real time data ingestion
- Created a Docker and Kubernetes based Hadoop cluster with opensource technologies in place of HortonWorks Hadoop cluster.
- Implemented code scan using Fortify on demand for vulnerabilities scan
- Configured monitoring system using Prometheus, Node-exporter and Grafana with alerting mechanism
- Configured Docker swarm and Kubernetes using Rancher to deploy high available containers and applications
- Built shell scripts to automate installations.
Developer Programmer,Client-Windstream
Prodapt Solutions
01.2015 - 12.2016
- Analyzed APTIS customer data and their database models
- Developed COBOL scripts to map source data to match with destination application data model
- Created test plan and test data for UAT.
- Built JCL jobs in dev and test for regression testing.
- Built script/scrub to load missed/dropped records.
- Application merge saved 2 millions/year for customer.
- Debugged complex software issues, leading to a more stable product release.
- Participated in regular code reviews, ensuring high-quality standards were consistently met across all development efforts.
- Automated repetitive tasks through scripting, freeing up valuable time for higher-priority projects.
Mainframe Developer, Client-CenturyLink
Prodapt Solutions
01.2012 - 09.2015
- Analyzed COBOL code to understand existing LPP (Late payment penalty) methods and document them.
- Enhanced COBOL code to change LPP charges based on customer type as per business requirements.
- Developed CICS code functionality to add additional screen on application and necessary changes to backend.
- Built data jobs (daily, weekly and monthly) to load data to Netezza data warehouse for analytics.
- Updated treatment techniques using additional COBOL modules for non-payment customers in CAT.
- Corrected/cleaned up DB2 records using scripts which have no relations.
- Generated reports for SwitchGate team to take care of uncompleted records.
- Built macros on application to make create relations on different customer devices and switches.
- Designed and developed robust COBOL programs, resulting in improved operational productivity.
Education
Bachelor of Technology - Computer Science and Engineering
Sri Venkateswara University
Tirupathi,India 05.2001 -
Skills
Proficient in data lake migration and modernizing application
Certification
Azure Administrator
Timeline
Google professional Architect
01-2024
Lead Data Engineer, Client- Verizon
Prodapt Solutions
08.2023 - Current
Lead Data Engineer, Client-Verizon
Prodapt Solutions
10.2022 - 07.2023
Google cloud Data enginner
06-2022
Technical Lead, Client-Verizon
Prodapt Solutions
04.2022 - 09.2022
Data Engineer,Client-Windstream
Prodapt Solutions
06.2021 - 03.2022
Graph Database Developer,Client-Windstream
Prodapt Solutions
08.2020 - 05.2021
Azure Devops expert
03-2020
Azure Administrator
01-2019
Bigdata DevOps Engineer,Client-Windstream
Prodapt Solutions
01.2017 - 07.2020
Developer Programmer,Client-Windstream
Prodapt Solutions
01.2015 - 12.2016
Mainframe Developer, Client-CenturyLink
Prodapt Solutions
01.2012 - 09.2015
Bachelor of Technology - Computer Science and Engineering
Sri Venkateswara University
05.2001 -