Study finds LLM agents favor items by source, impacting decisions

arXiv: arXiv is an open-access repository for scholarly articles primarily in physics, mathematics, computer science, and related fields. In the context of the study, LLM agents in scholarly search tasks consistently favor results from arXiv over other platforms even when relevance is equal. Medium: Medium is an online publishing platform hosting user-generated articles on technology, business, and various topics. The study indicates that LLM agents in scholarly contexts tend to disfavor results from Medium compared to established academic repositories. Reddit: Reddit is a social news aggregation and discussion website with communities covering diverse subjects including technology and research. The paper finds that LLM agents generally avoid citing or selecting content from Reddit in scholarly search scenarios relative to academic sources. Expedia: Expedia is a major online travel agency offering hotel bookings, flights, and vacation packages. The research shows that several LLM agent models avoid Expedia for hotel recommendations in favor of competitors like Booking.com when items are otherwise equivalent. YouTube: YouTube is a video-sharing platform with extensive educational and technical content. Research results show LLM agents in academic search tasks prefer other sources over YouTube even when content relevance is comparable. OpenReview: OpenReview is a platform for open peer review and hosting of conference papers, especially in machine learning and computer science. The study notes that LLM agents favor scholarly results from OpenReview in relevant searches. Booking.com: Booking.com is a leading online platform for hotel and accommodation reservations worldwide. According to the paper, most tested LLM agents prefer Booking.com as a source for hotel bookings over alternatives such as Expedia, even for identical listings. ACL Anthology: ACL Anthology is the digital archive of the Association for Computational Linguistics, providing access to NLP and computational linguistics papers. The paper highlights that LLM agents lean toward this source in scholarly search tasks over less formal platforms. Source Preference in the Wild: Source Preference in the Wild is a research paper examining how LLM agents exhibit biases toward items from specific sources during decision-making tasks like product selection, hotel booking, and scholarly search. The study analyzes 12 agent models across multiple domains and finds that source identification alone can override item quality in agent choices. It demonstrates that hiding source information or supplying missing details reduces this preference bias. Agent Decision Bias: LLM agents can select inferior options solely because they originate from a preferred source rather than evaluating item quality alone. Mitigation Approaches: Supplying missing details such as prices or using targeted prompts can reduce source-based preferences in LLM agent behavior. Source Influence Mechanism: Identifying information about an item's source directly affects selection, with hiding URLs weakening preferences and relabeling items with favored sources increasing their selection.