Building an AI 10-Q Analyzer: Part 3 | Evaluating Results and Insights using O1-mini -using 10Qs from Microsoft and Rigetti

Read Part 1 here.

Read Part 2 here.

In the dynamic world of financial analysis, the ability to swiftly and accurately interpret complex quarterly filings like the SEC’s Form 10-Q is invaluable. To address this need, I developed an AI-driven pipeline leveraging the Google/flan-t5-base model, Retrieval-Augmented Generation (RAG), and Named Entity Recognition (NER). Recently, I sought to evaluate the effectiveness of this pipeline using O1-mini, a trusted AI analyzer, focusing on two prominent companies: Microsoft and Rigetti. This blog post delves into the performance of my pipeline, highlighting its strengths, identifying areas for improvement, and sharing key takeaways that underscore the potential and challenges of AI in financial analysis.

Introduction to My 10-Q Analyzer Pipeline

Before diving into the evaluation, it’s essential to understand the components of my pipeline:

1. Google/flan-t5-base Model: A versatile language model fine-tuned for various NLP tasks, providing robust capabilities for text generation and summarization.
2. Retrieval-Augmented Generation (RAG): Enhances the model’s ability to retrieve relevant information from large datasets, ensuring that the analysis is both comprehensive and contextually accurate.
3. Named Entity Recognition (NER): Identifies and categorizes key financial entities such as revenue, expenses, net income, and other pertinent metrics within the text.

The culmination of these technologies is a powerful tool designed to parse, interpret, and summarize 10-Q filings, generating a structured JSON output that highlights critical financial data.

O1-mini’s Evaluation: Microsoft

High-Level Observations

O1-mini’s analysis of my pipeline’s performance on Microsoft’s 10-Q filing as of September 30, 2024, provided insightful feedback. The pipeline successfully condensed the extensive filing into a “ConsolidatedSummary,” extracting a diverse range of numeric values, including financial statement line items, dates, share information, and significant narrative references (e.g., the $75.4 billion purchase price for Activision).

Strengths Identified

1. Comprehensive Data Capture: The pipeline effectively identified key financial numbers such as net income and total revenue . This demonstrates the robustness of the NER component in capturing a wide spectrum of monetary values.
2. High Recall Rate: The system successfully extracted the majority of numeric references from the summary, indicating that the pipeline is adept at identifying relevant data points within complex financial documents.

Areas for Improvement

1. Label Mismatches: O1-mini noted instances where numeric values were incorrectly labeled. For example, figures related to “Total cost of revenue” were sometimes misclassified under “Regex_Revenue.” This suggests that while the extraction is thorough, the categorization needs refinement to ensure each numeric value is accurately labeled.
2. Duplicate and Partial Extracts: The pipeline occasionally extracted the same number multiple times under different labels or captured incomplete figures (e.g., “2314” from $22,314). This points to a need for better deduplication and validation mechanisms within the pipeline.
3. Contextual Misalignment: The system struggled to assign the correct financial categories, often grouping disparate items under incorrect labels. Enhancing contextual understanding could mitigate this issue, ensuring that each extracted value is placed within the appropriate financial category.

Overall Assessment

O1-mini rated the pipeline’s accuracy for Microsoft’s 10-Q at 60-70%. While the tool effectively identified relevant currency amounts, the precision in labeling and categorization was inconsistent. This feedback is invaluable, highlighting the importance of refining both the extraction patterns and the contextual analysis to enhance overall accuracy.

O1-mini’s Evaluation: Rigetti

Detailed Assessment

Turning to Rigetti’s Form 10-Q, the evaluation by O1-mini revealed both commendable strengths and notable shortcomings of the pipeline.

Strengths

1. Accurate Financial Data Extraction: The pipeline accurately captured key financial figures such as total assets, total liabilities, revenue, and net loss. This demonstrates the pipeline’s capability to extract precise numerical data from financial statements.
2. High-Level Coverage: The tool successfully identified standard 10-Q sections, including the cover page, table of contents, and financial statements, ensuring a comprehensive overview of the filing.

Weaknesses

1. Extraneous References: The summary included unrelated content, such as mentions of Enron North America Corp. and outdated disclaimers. This indicates gaps in the text-cleaning process, where irrelevant information from other documents inadvertently slips into the summary.
2. Repetitive and Scrambled Formatting: Sections appeared repeated or awkwardly merged, compromising the readability and coherence of the summary. This suggests that the pipeline needs improved formatting controls to ensure the output is well-structured and free from redundancies.
3. Inconsistencies in Headings and Content: Headings often merged with content from other sections, leading to confusion and a lack of clear organization within the summary.

Accuracy Score

O1-mini assigned an overall accuracy rating of 70-75% for Rigetti’s 10-Q. The numerical data extraction was largely accurate, but the narrative text suffered from significant clutter and irrelevant references, highlighting the need for enhanced text processing and contextual understanding.

Specific Observations

• Correct Financial Data: The pipeline successfully extracted balance sheet and income statement figures that matched Rigetti’s actual Q3 2024 results, underscoring the effectiveness of the NER component in numerical data extraction.
• Incorrect or Extraneous Mentions: Erroneous content, such as references to Enron and outdated disclaimers, pointed to deficiencies in the text-cleaning mechanisms. Implementing more robust filtering and validation steps could address these issues.
• Formatting & Redundancy: The summary resembled a raw text dump with repeated sections and spliced disclaimers, rather than a coherent and polished overview. Enhancing the pipeline’s ability to structure the output logically is crucial for improving readability and usability.

Recommendations and Future Enhancements

Based on O1-mini’s evaluation, several actionable recommendations emerge to enhance the pipeline’s performance:

1. Enhanced Text-Cleaning Pipelines: Implement more robust parsing techniques to eliminate unrelated disclaimers and ensure only relevant content is included. This could involve developing more sophisticated filters or leveraging additional NLP techniques to better distinguish between relevant and irrelevant text.
2. Improved Labeling Accuracy: Refine the regex patterns and NER configurations to ensure that each numeric value is accurately labeled according to its financial category. Incorporating contextual analysis can help in correctly classifying figures under “Revenue,” “Operating Income,” “Net Income,” etc.
3. Deduplication and Validation Mechanisms: Introduce steps to identify and remove duplicate or partial extracts, ensuring that each financial metric is captured once and accurately.
4. Structured Summarization: Develop mechanisms to produce a curated summary that distills key financial highlights and major forward-looking statements without extraneous clutter. This could involve leveraging advanced summarization techniques or integrating additional layers of validation to ensure coherence.
5. Artifact Removal: Address and remove residual artifacts from other documents (e.g., Enron references) that erroneously appear in the summary. This might involve training the model to better recognize and exclude such anomalies or implementing post-processing checks to filter out unrelated content.

Conclusion

The evaluation of my AI-powered 10-Q analyzer pipeline using O1-mini’s assessment of Microsoft and Rigetti’s filings has provided invaluable insights into its current performance and areas for improvement. While the pipeline demonstrates strong capabilities in extracting key financial data with high recall, challenges remain in ensuring precise labeling, eliminating duplicates, and maintaining coherent and relevant summaries.

Final Takeaway: My 10-Q analyzer pipeline is a promising tool for capturing a wide range of financial metrics from SEC filings, offering substantial benefits in terms of efficiency and breadth of data extraction. However, to fully realize its potential, further refinements are necessary to enhance label accuracy, eliminate extraneous content, and ensure the output is both precise and user-friendly. By addressing these areas, the pipeline can evolve into a more reliable and insightful tool for financial analysts and stakeholders, paving the way for more advanced AI-driven financial analysis solutions.

For those interested in exploring the project further, feel free to check out the GitHub repository.