AI Data Engineer
Open To Work
ManiKanta Reddy
Capital One
Pittsburgh,PA
7
years of professional experience
Results-driven AI Data Engineer with 6+ years in Data Engineering and Cloud Data Platforms. Expertise in designing scalable ETL/ELT pipelines and AI applications using Python, SQL, and Apache Spark. Specializes in LLMs, RAG, and AI solutions, focusing on production readiness. Holds a Master's Degree in Information Systems Management, committed to advancing data-driven insights and operational efficiency.
Work History
1 Year 8 Months
03.2025 - Current
1 Year 8 Months
AI Data Engineer
Capital One | Pittsburgh, Pennsylvania
03.2025 - Current
- Developed RAG workflows using Amazon Bedrock, LangChain, and vector databases to retrieve relevant information from internal documents and provide context to large language models.
- Built Python-based ingestion pipelines to collect structured and unstructured data from Amazon S3, relational databases, and internal APIs for use in AI applications.
- Created document-processing jobs to extract text, split large files into smaller chunks, and generate embeddings for efficient storage and searchability of content with required metadata.
- Worked on AI agent workflows using LangGraph to connect LLMs with approved tools and APIs, allowing users to complete multi-step information retrieval tasks.
- Developed FastAPI endpoints to make AI services available to internal applications, including request validation, authentication, and error handling.
- Used AWS Glue, PySpark, and SQL to clean and transform source datasets before making them available to AI and analytics workflows.
- Worked with Docker, Git, and CI/CD pipelines to test, package, and deploy Python services across development and production environments.
1 Year 1 Month
01.2024 - 02.2025
1 Year 1 Month
Data Engineer
CVS Healthcare | Pittsburgh, Pennsylvania
01.2024 - 02.2025
- Developed Azure Data Factory pipelines to ingest claims, pharmacy, and member data from relational databases, flat files, and external data sources into ADLS Gen2.
- Used Azure Databricks, PySpark, and Spark SQL to clean incoming records, standardize data formats, and apply business transformation rules.
- Implemented incremental data loading with watermark columns and Delta Lake MERGE statements, enhancing efficiency by processing new and updated records without full table reloads.
- Organized data into Bronze, Silver, and Gold layers using Delta Lake, improving clarity and accessibility of validated and reporting-ready datasets.
- Worked on Spark performance issues by reviewing job execution details, adjusting partitioning, and improving joins used in large data transformations.
- Added data quality checks for missing values, duplicate records, schema changes, and source-to-target record counts, ensuring reliability of published processed data.
- Used Git and Azure DevOps to manage code changes, review pull requests, and support deployments between development and production environments.
1 Year 4 Months
03.2022 - 07.2023
1 Year 4 Months
Analyst Engineer
American Express | Hyderabad, Telangana
03.2022 - 07.2023
- Developed SQL queries and stored procedures to efficiently process transaction records from various source tables, preparing accurate datasets for financial reporting.
- Reviewed and optimized slow-running SQL queries by modifying joins, filtering conditions, and query logic, enhancing database processing performance.
- Worked on ETL transformations to standardize transaction dates, reference numbers, status codes, and other fields before loading data into reporting tables.
- Used CTEs, window functions, and joins to investigate duplicate transactions, missing records, and differences between source and reporting systems.
- Created Python and Pandas scripts to automate recurring reconciliation checks and reduce manual comparisons performed in Excel.
- Investigated failed batch jobs by reviewing execution logs, checking source files, and identifying issues caused by missing or incorrectly formatted data.
- Prepared exception reports to help operations teams identify transaction records requiring additional review or correction.
- Coordinated with developers, QA teams, and business analysts to rigorously test data-processing changes, ensuring results met specified business requirements.
- Maintained SQL scripts, source-to-target mapping documents, and reconciliation procedures used during production support and troubleshooting.
1 Year 11 Months
03.2020 - 02.2022
1 Year 11 Months
Data Analyst
Tata Consultancy Services (TCS) | Hyderabad, Telangana
03.2020 - 02.2022
- Created Power BI dashboards to track operational KPIs, compare monthly performance, and identify changes in business metrics.
- Developed SQL queries with joins, aggregations, subqueries, and filters to extract and prepare data from relational databases for insightful business analysis and reporting.
- Prepared daily, weekly, and monthly reports using SQL and Excel, including Pivot Tables, VLOOKUP, and other formulas for data analysis.
- Collaborated with business analysts to gather reporting requirements and updated SQL queries and Power BI dashboards to enhance accuracy and relevance.
- Validated Power BI reports against SQL results to confirm that dashboard figures matched the underlying datasets.
- Compared data between source systems and reporting tables to investigate differences in record counts and KPI calculations.
- Streamlined dataset cleaning by identifying and resolving duplicate records, missing values, incorrect formats, and inconsistencies across source files.
- Used Python and Pandas for basic data cleaning, CSV processing, and automating repetitive reporting activities.
- Documented report definitions, SQL logic, and recurring data issues to support future updates and troubleshooting.
Education
Master of Science - Management Information Systems
Robert Morris University | Coraopolis, PA | 05-2025
Skills
Programming Languages: Python
SQL
PySpark
Scala
Bash/Shell Scripting
Generative AI & LLMs: Generative AI
Large Language Models (LLMs)
Retrieval-Augmented Generation (RAG)
Timeline
AI Data Engineer
Capital One
03.2025 - CurrentRead more
Data Engineer
CVS Healthcare
01.2024 - 02.2025Read more
Analyst Engineer
American Express
03.2022 - 07.2023Read more
Data Analyst
Tata Consultancy Services (TCS)
03.2020 - 02.2022Read more
Robert Morris University - Master of Science, Management Information Systems
Read more