Call for Papers: Workshop on The Use of Generative AI for Science of Science and Higher Education Studies (AISci)

Submission deadline: 31st of August 2024

The development of generative AI language models, i.e. “a class of artificial intelligence models that can create new data based on patterns and structures learned from existing data” (​Ray 2023​) represents one of the most significant technological change of the recent years. The development of the models has sparked wide interest in the research community, but also in various application fields of research results. The versatility of such models, such as ChatGPT developed by OpenAI, makes them suitable for a wide range of scientific research and its application. The literature highlights both risks and challenges of the use of generative AI including increased efficiency and accuracy, but also security concerns and capability limitations (​Deng & Lin, 2022​). 

As for the impact of generative AI on research and higher education, the literature on AI focuses first and foremost on the implications for educational activities (​Grassini, 2023​), including specifically its usage for scientific writing (​Imran & Almusharraf, 2023​). Potential issues in terms of quality and ethical concerns (e.g., ChatGPT does not cite its sources) have been highlighted in this context (​Haman & Školník, 2023​). 

Generative AI has some attractive features for science and higher education studies, including the fact that it can access and summarise a wide range of sources available on the Internet. The interactive features of ChatGPT are potentially relevant as these allow users to interact with the AI to get more precise and relevant results (​Bornmann & Lepori, 2024). 

In scientometrics, ChatGPT has been used in some recent studies such as to identify research topics (​Tang, Zhou & Lu, 2023​), to predict citation counts (​de Winter, 2024​) and to identify prominent scholars (​Sandnes, 2024​). Results regarding the usefulness of AI in that respect are somewhat mixed, with some studies highlighting the potential of generative AI, while others argue that the quality of results falls short of the current standards in scientometrics (​Sandnes, 2024​). Since the evolution of language models is so rapid, these research results from the empirical studies may be outdated soon. Three other areas which have been investigated in scientometrics are the use of ChatGPT to evaluate research quality (​Thelwall, 2024​), to identify similar institutions for benchmarking (​Bornmann & Lepori, 2024​) and to categorise research papers based on metadata of publications such as titles and abstracts (​van Eck & Waltman, 2024​). In these areas, we expect that the interactive nature and easy use of ChatGPT will become highly attractive to (non-)expert users. Non-expert users frequently do not want to bother with the use of more complex tools requiring expert knowledge and interpretation. Expert users are interested in the potential of the new competitor. 

Since many areas in scientometrics have been and will be more and more confronted with the use of generative AI, the goal of the workshop is to advance our understanding of its potential, including but not limited to ChatGPT. The workshop is intended to identify pitfalls and risks for the science of science and higher education studies and to propose approaches to mitigate them. 

References

Bornmann, L., & Lepori, B. (2024). The use of ChatGPT to find similar institutions for institutional benchmarking. Scientometrics, 1-6. 

de Winter, J. (2024). Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstracts. Scientometrics, 1-19. 

Deng, J., & Lin, Y. (2022). The benefits and challenges of ChatGPT: An overview. Frontiers in Computing and Intelligent Systems, 2(2), 81-83. 

Grassini, S. (2023). Shaping the future of education: exploring the potential and consequences of AI and ChatGPT in educational settings. Education Sciences, 13(7), 692. 

Haman, M., & Školník, M. (2023). Using ChatGPT to conduct a literature review. Accountability in research, 1-3. 

Imran, M., & Almusharraf, N. (2023). Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4), ep464. 

Ray, P. P. (2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems. 

Sandnes, F. E. (2024). Can we identify prominent scholars using ChatGPT?. Scientometrics, 129(1), 713-718. 

Tang, L., Zhou, X., & Lu, M. (2023). A GPT-Based Approach for Scientometric Analysis: Exploring the Landscape of Artificial Intelligence Research. arXiv preprint arXiv:2304.09487. 

Thelwall, M. (2024). Can ChatGPT evaluate research quality?. arXiv preprint arXiv:2402.05519. 

van Eck, N. J., & Waltman, L. (2024, January 24). An open approach for classifying research publications. Leiden Madtrics. https://www.leidenmadtrics.nl/articles/an-open-approach-for-classifying-...  

Contributions

The organisers of the AISci workshop particularly welcome empirical studies comparing systematically the output from generative AI (e.g., ChatGPT) with more standard approaches in scientometrics, science of science studies, and higher education studies to perform tasks such as: 

• The labelling of research topics based on publication metadata. 

• The identification of prominent scholars and the measurement of their performance. 

• The evaluation of research quality at the paper, individual and institutional level. 

• The analysis of scientific careers (e.g., the identification of certain structures). 

• The identification of similar institutions in benchmarking studies. 

• The characterisation of universities’ missions, strategies and organisation. 

Workshop organisation 

The AISci workshop will be organised in a set of thematic sessions around the topics introduced above. Each accepted submission will be assigned to a discussant, who will be expected to read the paper in advance and provide detailed feedback. 

Funding 

A limited number of grants will be offered to cover travel and accommodation, particularly for young researchers. 

Reviewing process 

We welcome submissions presenting ongoing work and including at least some preliminary results. Short papers of around 3000 words (incl. references, appendices and other material) with a short abstract of 50-150 words should be submitted, using the Paper Template provided, via EasyChair (https://easychair.org/conferences/?conf=1gasshs) by 31st August 2024. Short papers must be exclusively in English and include the name and e-mail address of the presenter and the title of the contribution. 

All contributions submitted to the conference will be reviewed by at least two anonymous reviewers. Based on the outcome of the peer review process, the conference’s programme committee will select the contributions for presentation at the conference, either as an oral presentation or as a poster presentation. Submissions will be peer-reviewed and selected based on their quality. 

Publication 

Selected papers, as well as the main conclusions of the workshop, will be published in the journal Scientometrics. 

Scientific committee: Benedetto Lepori (Università della Svizzera italiana), Lutz Bornmann (Max Planck Society), Frans Kaiser (University of Twente), Wolfgang Glänzel (KU Leuven), Marek Kwiek (University of Poznan), Mike Thelwall (University of Sheffield), Marco Steenbergen (University of Zurich), Jens Peter Andersen (Aarhus University)

Organisation committee: Agata Lambrechts (Università della Svizzera italiana), Pınar Eldemir (Università della Svizzera italiana).  

Important dates: