Exploring ChatGPT in Academic Information Retrieval: A Critical Review

Objective. To compare scientific search behavior with and without ChatGPT assistance. Design/Method. Comparative study with two randomly assigned groups of graduate students; five tasks based on cognitive levels. Tools/Settings. ChatGPT for Google extension version 3.5 and web search; data collected via screen recording. Metrics. Number of searches, query length, SERP clicks, SERP browsing time, number/time of URL visits, total retrieval time. Key Findings. Reduced clicks, reduced number/time of URL visits, and reduced total retrieval time with ChatGPT; increased query length and SERP browsing time; significant differences mainly in T1, T2, and T5. Conclusion. ChatGPT excels in recall, understanding, and creation tasks; limited effect on analysis and evaluation.