Jegan Gopal
Plano,TX
Summary
Experienced AWS Data Engineer with over 14 years of dedicated expertise in designing, implementing, and optimizing data solutions on the Amazon Web Services (AWS) platform. Proficient in leveraging a diverse skill set encompassing data architecture, ETL processes, data warehousing, and analytics to drive actionable insights and enhance business intelligence capabilities. Demonstrated track record of delivering high-quality, scalable data pipelines and solutions that meet the evolving needs of diverse industries. Adept at collaborating with cross-functional teams to translate business requirements into technical solutions, ensuring alignment with strategic objectives and delivering measurable value. Proven ability to stay abreast of emerging technologies and best practices in the rapidly evolving field of cloud-based data engineering, while maintaining a focus on innovation and continuous improvement. Dedicated to driving organizational success through the effective utilization of data as a strategic asset.
Overview
13
13
years of professional experience
1
1
Certification
Work History
Senior Data Engineer
Cognizant Technology Solutions, CTS
Plano, TX
06.2021 - Current
- Spearheaded the design and implementation of a Big Data Lake/Hub infrastructure project on AWS Cloud, facilitating the storage and processing of large volumes of data.
- Utilized the Hadoop Ecosystem technologies including EMR, ECS, EKS, HDFS, YARN, MAP Reduce, Hive, Sqoop, and Apache Spark with Scala Programming to develop scalable and efficient data processing solutions.
- Participated in architectural discussions to build confidence and promote customer success when developing new and migrating existing applications, software and services on AWS platform.
- Designed and deployed complex applications within the data lake/data hub environment, ensuring high availability, reliability, and performance.
- Monitored and optimized functionality performance using AWS CloudWatch, Lambda Function, and Airflow, scheduling and managing jobs efficiently.
- Leveraged Python scripting to build custom scripts for automating workflows and deploying Airflow pipelines in Jenkins.
- Demonstrated proficiency in handling various file formats of semi-structured data such as TXT, CSV, Parquet, ORC, JSON, Avro, and XML.
- Managed diverse databases including SQL Server, OpenEdge, Db2 for IBM, Db2 for LUW, Oracle, PostgreSQL, Athena, MySQL, MariaDB, SQLite, Apache Hive, Apache Ignite, Apache Kylin, Azure SQL Server, Google BigQuery, and Teradata.
- Automated daily backups of critical data using AWS S3, reducing the risk of data loss in case of system failures.
- Collaborated closely with cross-functional teams to define project requirements, develop end-to-end solutions, and ensure successful project delivery.
- Evaluated emerging technologies and tools to identify opportunities for enhancing existing systems and creating new ones, keeping the infrastructure up-to-date with industry standards.
- Reengineered existing ETL workflows to improve performance, identifying bottlenecks and implementing optimizations.
- Ensured data quality through rigorous testing, validation, and monitoring of all data assets, minimizing inaccuracies and inconsistencies.
- Mentored junior team members, sharing best practices in software development, code optimization, and troubleshooting techniques.
- Participated in strategic planning sessions with stakeholders to assess business needs related to data engineering initiatives and provide strategic insights for project success.
- Collaborated with cross-functional teams to define requirements and develop end-to-end solutions for complex data engineering projects.
- Evaluated emerging technologies and tools to identify opportunities for enhancing existing systems or creating new ones.
- Designed custom monitoring dashboards using Amazon CloudWatch for real-time insight into system health and performance metrics.
- Facilitated seamless integration between on-premise systems and AWS resources, streamlining workflows across organizational boundaries.
- Developed comprehensive strategies for leveraging AWS Auto Scaling, ensuring optimal resource allocation and cost efficiency during periods of high demand.
- Developed and maintained CI/CD pipelines using Jenkins, increasing deployment speed and reliability.
Senior Software Developer
SPi Technologies India Private Limited
Chennai, India
01.2021 - 06.2021
- Collaborated with stakeholders to define project scope, objectives, and deliverables, ensuring alignment with organizational goals and requirements.
- Designed and developed the data warehouse architecture, including data modeling, schema design, and ETL processes, to facilitate efficient data storage, retrieval, and analysis.
- Implemented robust ETL workflows using tools such as Informatica, SSIS, or Apache Nifi to extract, transform, and load data from various source systems into the data warehouse.
- Ensured data quality and integrity through rigorous testing, validation, and monitoring of data pipelines, minimizing inaccuracies and inconsistencies.
- Optimized data warehouse performance by fine-tuning SQL queries, indexing strategies, and partitioning techniques to improve data retrieval and processing speed.
- Implemented data governance policies and procedures to ensure compliance with regulatory requirements and industry best practices, maintaining data security and confidentiality.
- Developed interactive dashboards and reports using BI tools such as Tableau, Power BI, or QlikView to provide stakeholders with actionable insights into key business metrics and trends.
- Conducted regular performance reviews and optimization efforts to identify areas for improvement and implement enhancements to meet evolving business needs.
- Provided training and support to end-users on data warehouse usage, reporting tools, and data analysis techniques to promote self-service analytics and decision-making across the organization.
- Acted as a liaison between business users, data analysts, and IT teams to gather requirements, prioritize tasks, and ensure successful project delivery within established timelines and budget constraints.
Senior Software Engineer
Newgen Knowledge Works
Chennai, India
03.2019 - 01.2021
- Developed scalable applications using agile methodologies for timely project delivery.
- Optimized code performance for improved user experience and system efficiency.
- Managed multiple projects simultaneously while maintaining strict deadlines and high-quality standards.
- Played a pivotal role as a Data Warehouse and BI Analyst, overseeing the design, development, and maintenance of data warehousing solutions.
- Collaborated with stakeholders to gather requirements and define project scope, ensuring alignment with business objectives.
- Designed and implemented scalable data warehouse solutions, leveraging technologies such as SQL Server, Oracle, PostgreSQL, Snowflake, and Azure SQL Database.
- Developed ETL/ELT processes using tools like Informatica, Integration Services (SSIS), and Apache Airflow to extract, transform, and load data from various source systems.
- Conducted thorough data analysis and profiling to ensure data integrity, quality, and consistency across the data warehouse.
- Created and maintained dimensional data models (star schema/snowflake schema) to support business reporting and analytics requirements.
- Developed interactive dashboards and reports using BI tools like Tableau, Power BI, and SAP BusinessObjects, providing actionable insights to stakeholders.
- Participated in data governance initiatives, defining data standards, policies, and procedures to ensure compliance and data security.
- Provided support and troubleshooting for data warehouse and BI solutions, resolving issues promptly to minimize downtime and ensure system reliability.
- Collaborated with cross-functional teams including business analysts, developers, and QA testers to deliver end-to-end data solutions.
- Conducted user training sessions and provided documentation to facilitate user adoption of BI tools and reports.
- Stayed abreast of emerging trends and best practices in data warehousing, BI, and analytics, recommending and implementing innovative solutions to enhance data capabilities and drive business value.
Software Developer
Lumina Datamatics Ltd
Chennai, India
01.2016 - 03.2019
- Served as a BI Analyst within a legacy system .NET application project, responsible for data analysis, modeling, and reporting.
- Worked closely with stakeholders to understand business requirements and translate them into technical solutions within the .NET framework.
- Conducted data profiling and analysis to identify trends, patterns, and anomalies in the data, providing actionable insights to business users.
- Developed and maintained ETL processes using tools like SSIS (SQL Server Integration Services) and Informatica to extract, transform, and load data from disparate sources into the data warehouse.
- Designed and implemented dimensional data models (star schema/snowflake schema) to support business reporting and analytics needs.
- Created interactive reports, dashboards, and data visualizations using BI tools such as Microsoft Power BI, SQL Server Reporting Services (SSRS), and Tableau.
- Collaborated with developers to enhance existing .NET applications with embedded analytics and reporting capabilities, ensuring seamless integration of BI components.
- Conducted user acceptance testing (UAT) and provided end-user training and support to facilitate the adoption of BI solutions.
- Participated in system upgrades and migrations, ensuring data integrity and compatibility with legacy systems.
- Maintained documentation of BI solutions, including data dictionaries, technical specifications, and user guides.
- Proactively identified opportunities for process improvements and automation to optimize BI workflows and enhance productivity.
- Stayed updated with industry trends and best practices in BI, data analytics, and .NET development, continuously enhancing skills and knowledge to drive project success.
- Handled high volumes of incoming work requests professionally while maintaining excellent organization.
- Collaborated with team members for enhanced process efficiency and productivity.
- Managed multiple tasks simultaneously, ensuring timely completion of all assignments.
- Communicated effectively with colleagues and supervisors to resolve issues promptly and accurately.
Software Developer
Capstone Solution
Chennai, India
09.2011 - 11.2015
- Developed and maintained web applications using ASP.NET MVC, C#, and .NET Framework.
- Designed and implemented scalable and efficient database schemas using SQL Server, optimizing performance and reliability.
- Maintained existing software systems by identifying and correcting software defects.
- Led the transition to microservices architecture, improving system scalability and maintainability.
- Collaborated with cross-functional teams including designers, testers, and product managers to deliver high-quality software solutions within project timelines.
- Utilized front-end technologies such as HTML5, CSS3, JavaScript, and jQuery to enhance user experience and interactivity.
- Integrated third-party APIs and services to extend application functionality and improve user workflows.
- Implemented security measures such as data encryption, authentication, and authorization to ensure compliance with industry standards and protect sensitive information.
- Participated in code reviews, debugging sessions, and performance tuning to maintain code quality and optimize application performance.
- Resolved technical issues and provided timely support to end-users, troubleshooting and diagnosing software problems to minimize downtime and ensure business continuity.
- Actively engaged in learning new technologies and staying updated with industry trends to enhance technical skills and contribute innovative solutions to development projects.
Education
High School Diploma -
Kalaivanar NSK Polytechnic College
Tamilnadu, India04.2010
Skills
Big Data Processing
Lambda Functions
Data Lake Management
Performance Optimization
Amazon S3 Proficiency
Hadoop Ecosystem Familiarity
ETL Design and Implementation
Real-time Data Streaming
Cloud Architecture Design
Load Balancing
Monitoring tools
Scripting Languages
Cloud Computing
Firewalls
Python Programming
Git Version Control
ETL development
Kafka Streaming
Data Pipeline Design
Data Modeling
API Development
Hadoop Ecosystem
Performance Tuning
Data Warehousing
Spark Development
Data Security
Machine Learning
Data Quality Assurance
Scala Programming
SQL and Databases
Database Design
Data Migration
Data Analysis
Certification
- AWS Certified Cloud Practitioner (CLF)
- Snowflake
- SQL Expert
- The Hadoop Ecosystem Master Class
- Big Data University
- Python
- Spark Scala
Timeline
Senior Data Engineer
Cognizant Technology Solutions, CTS
06.2021 - Current
Senior Software Developer
SPi Technologies India Private Limited
01.2021 - 06.2021
Senior Software Engineer
Newgen Knowledge Works
03.2019 - 01.2021
Software Developer
Lumina Datamatics Ltd
01.2016 - 03.2019
Software Developer
Capstone Solution
09.2011 - 11.2015
High School Diploma -
Kalaivanar NSK Polytechnic College