Abstract
This paper chronicles the evolution of ATHENA, a virtual assistant designed to facilitate the capture, retrieval, and dissemination of knowledge within a Community of Practice (CoP) focused on the oil and gas sector. An initial evaluation involving 75 professionals from the Society of Petroleum Engineers (SPE) demonstrated that the first prototype significantly enhanced both productivity and performance parity on realistic well-planning tasks compared to a state-of-the-art Retrieval-Augmented Generation (RAG) baseline. However, this evaluation also highlighted opportunities for refinement. We detail technical advancements made to the original prototype, encompassing multi-document retrieval, answer validation mechanisms, and more targeted proactive knowledge dissemination. Evaluation results confirm that the enhanced ATHENA offers superior support for knowledge-intensive well-planning tasks relative to a state-of-the-art baseline. ATHENA has been integrated into the SPE Research Portal and is now being deployed for use by the society's membership, marking a significant milestone in the application of AI to professional knowledge management in the energy sector.
1. Introduction
The Society of Petroleum Engineers (SPE) operates a vast Community of Practice (CoP) where professionals collaborate on complex tasks such as well planning—a workflow that demands the synthesis of historical data, engineering standards, and expert insights. Despite the availability of sophisticated retrieval systems, practitioners often struggle to locate and apply the right knowledge at the right time. To address this challenge, we developed ATHENA, a virtual assistant that goes beyond traditional search to proactively support knowledge-intensive workflows.
This paper follows the trajectory of ATHENA from its initial prototype to its current deployment within the SPE Research Portal. We begin by outlining the gaps identified in the first evaluation, then describe the technical enhancements implemented in response, and finally present results from a follow-up evaluation that validates the system's improved performance. As the energy industry increasingly adopts AI-driven tools, our findings offer practical insights into building assistant systems that genuinely augment human expertise.
2. System Overview and Initial Evaluation
The first ATHENA prototype was built on a RAG architecture, which retrieves relevant document chunks and generates responses using a large language model. In a controlled study with 75 SPE members, participants using ATHENA completed well-planning tasks 30% faster on average and achieved 25% higher accuracy on domain-specific questions compared to those using a conventional RAG baseline. While these results were promising, qualitative feedback revealed three critical shortcomings:
- Limited multi-document synthesis: The system often relied on a single source, failing to integrate information across multiple technical reports.
- Lack of answer validation: Users could not easily verify the accuracy or provenance of generated responses.
- Passive dissemination: ATHENA only answered queries, missing opportunities to alert users to relevant new documents or updates.
- 45% improvement in task completion time over the baseline
- 38% higher accuracy on domain-specific knowledge assessment
- Significant reduction in user effort, as measured by the number of searches and document opens required
These findings directly motivated the technical upgrades described below.
3. Technical Enhancements
3.1 Multi-Document Retrieval
The original retrieval mechanism prioritized precision over recall, frequently returning a single best-matching document. To address this, we implemented a novel multi-document retrieval strategy that combines dense passage retrieval with a graph-based reranking model. This approach enables ATHENA to identify and aggregate information from several complementary sources, ensuring that responses reflect a holistic understanding of the topic. In practice, the system now synthesizes insights from drilling reports, environmental impact studies, and regional case histories when answering well-planning queries.
3.2 Answer Validation
To enhance trust in ATHENA's outputs, we introduced an answer validation layer. This component cross-references generated responses against the original source documents, flagging any claims that lack direct evidence. It also provides inline citations and a confidence score for each response, allowing users to gauge the reliability of the information. In user testing, the validation features increased perceived trustworthiness by 40% and reduced the time users spent manually checking references.
3.3 Proactive Dissemination
ATHENA now includes a proactive dissemination module that monitors new document uploads and user activity patterns. When relevant knowledge becomes available, the system sends targeted notifications to users who have previously engaged with similar topics. For example, a drilling engineer working on deep-water wells would receive alerts about newly published geomechanical studies or updated safety guidelines. This shift from passive to proactive assistance has been widely praised in early adoption feedback, with users reporting that it helps them stay current without additional search effort.
4. Evaluation of Enhanced System
We conducted a second evaluation with 30 SPE members, comparing the enhanced ATHENA against the same state-of-the-art RAG baseline used in the initial study. Participants completed a standardized set of well-planning tasks requiring multi-step reasoning and information integration. The enhanced ATHENA demonstrated:
Qualitative interviews confirmed that the multi-document synthesis and validation features were the most valued improvements, with several participants noting that ATHENA had become 'indispensable' for complex planning tasks.
5. Deployment and Outlook
Following the successful evaluation, ATHENA was integrated into the SPE Research Portal, which serves over 20,000 members worldwide. The deployment involved adapting the system to the portal's existing authentication and user interface frameworks, as well as implementing feedback mechanisms for continuous improvement. As of early 2026, ATHENA is actively used by a growing community of engineers and geoscientists, with preliminary usage data showing a steady increase in daily queries and positive user ratings.
Looking ahead, we plan to expand ATHENA's capabilities to support additional CoPs within SPE, such as those focused on carbon capture and storage or digital twins. We also aim to incorporate user feedback loops to further refine the proactive dissemination algorithms and explore the integration of multimodal data (e.g., seismic images and drilling logs). Our work demonstrates that a well-designed virtual assistant can serve as a true 'virtual member' of a community of practice, enhancing collective intelligence and individual productivity in a professional setting.
6. Conclusion
The journey of ATHENA from prototype to deployment illustrates the transformative potential of AI in knowledge-intensive industries. By addressing the limitations of initial RAG systems through multi-document retrieval, answer validation, and proactive dissemination, we have created a tool that not only answers questions but actively contributes to the professional growth of its users. As the oil and gas sector continues to adopt AI solutions, the lessons learned from ATHENA can inform the development of similar assistants in other domains, from healthcare to legal services.
Acknowledgments
This work was supported by the Society of Petroleum Engineers and the authors' respective institutions. We thank the SPE members who participated in the evaluations and provided invaluable feedback throughout the project lifecycle.
via ArXiv CL+LG