doi: 10.56294/dm202214

 

ORIGINAL

 

Psychometric properties of an instrument to assess the level of knowledge about artificial intelligence in university professors

 

Propiedades psicométricas de un instrumento para evaluar el nivel de conocimiento sobre inteligencia artificial en docentes universitarios

 

Camilo Andrés Silva-Sánchez1  *

 

1Universidad Andrés Bello, Facultad de Enfermería. Santiago de Chile, Chile.

 

Cite as: Silva-Sánchez CA. Psychometric properties of an instrument to assess the level of knowledge about artificial intelligence in university professors. Metaverse Basic and Applied Research. 2022;1:14. https://doi.org/10.56294/mr202214

 

Submitted: 10-11-2022                      Revised: 29-11-2022                          Accepted: 23-12-2022                      Published: 26-12-2022

 

Editor: Lic. Mabel Cecilia Bonardi

 

ABSTRACT

 

Introduction: knowledge about AI in university professors allows them to integrate these technological tools into their teaching strategies and improve the quality of learning.

Objective: to determine the sustainable factorial structure of the relationship dimension of an instrument to evaluate the level of knowledge about artificial intelligence in university professors.

Methods: a cross-sectional metric validation study was conducted. A sample of 83 university professors was selected. An instrument on artificial intelligence for university professors was applied, consisting of 15 questions divided into three sections. Psychometric analysis was carried out to evaluate its validity and reliability.

Results: the results show that Part 1 has an alpha coefficient of 0,77, Part 2 has an alpha coefficient of 0,65, and Part 3 has an alpha coefficient of 0,83. The alpha coefficients for each subscale (0,77 for Part 1, 0,65 for Part 2, and 0,83 for Part 3) indicate that the instrument has good internal consistency and that the questions within each subscale are related to each other. The χ2/gl ratio of 2,1 indicates a good fit of the model, and the GFI, NFI, and CFI values are close to 1, indicating a good fit of the model.

Conclusions: the results of the present study support the validity, reliability, and sustainable factorial structure of the instrument on artificial intelligence for university professors, making it an appropriate tool to evaluate the level of knowledge about AI in university professors.

 

Keywords: Artificial Intelligence; Psychometric Properties; Reliability; Validity; University Professors.

 

RESUMEN

 

Introducción: el conocimiento sobre IA en los docentes universitarios les permite integrar estas herramientas tecnológicas en sus estrategias de enseñanza y mejorar la calidad del aprendizaje.

Objetivo: determinar la estructura factorial sustentable de la dimensión de relaciones de un instrumento para evaluar el nivel de conocimiento sobre inteligencia artificial en docentes universitarios.

Métodos: se realizó un estudio de validación métrica de corte transversal. Se seleccionó una muestra de 83 docentes universitarios. Se aplicó un instrumento sobre inteligencia artificial para docentes universitarios, que consta de 15 preguntas divididas en tres secciones. Se llevó a cabo un análisis psicométrico para evaluar su validez y fiabilidad.

Resultados: los resultados muestran que la parte 1 tiene un coeficiente alfa de 0,77, la parte 2 tiene un coeficiente alfa de 0,65 y la parte 3 tiene un coeficiente alfa de 0,83. Los coeficientes alfa para cada subescala (0,77 para la parte 1, 0,65 para la parte 2 y 0,83 para la parte 3) indican que el instrumento tiene una buena consistencia interna y que las preguntas dentro de cada subescala están relacionadas entre sí. La razón χ2/gl de 2,1 indica un buen ajuste del modelo, y los valores de GFI, NFI y CFI son cercanos a 1, lo que indica un buen ajuste del modelo.

Conclusiones: los resultados del presente estudio respaldan la validez, confiabilidad y estructura factorial sustentable del instrumento sobre inteligencia artificial para docentes universitarios, lo que la convierte en una herramienta adecuada para evaluar el nivel de conocimiento sobre IA en docentes universitarios.

 

Palabras clave: Inteligencia Artificial; Propiedades Psicométricas; Confiabilidad; Validez; Docentes Universitarios.

 

 

 

INTRODUCTION

Artificial intelligence (AI) is a branch of computer science that seeks to develop systems able to learn and carry out tasks requiring human intelligence, such as decision making, recognition of patterns, and natural language processing. At present, AI is transforming diverse sectors, including education, where it is opening new possibilities to improve the quality and efficiency of learning.(1)

University professors play a crucial role in training the new generations of professionals and in updating practicing professionals’ knowledge. It is important that these professors are familiar with the emerging technologies, like AI, so that they can prepare students for the future challenges and for being up-to-date in their own professional development.(2)

University professors’ knowledge of AI allows them both to integrate these technological tools into their teaching strategies and to improve the quality of learning. In this sense, AI can be used to evaluate the performance of students, which makes it possible to identify areas for improvement and to customize learning according to the individual needs of every student.(3)

AI can also be used in scientific research, which can improve research quality and facilitate the identification of patterns and relations in big datasets. The capacity of AI to process and analyze large numbers of data can also help university professors customize learning and teaching according to the individual needs of every student.(4)

Likewise, AI can be used to automate administrative and managerial tasks, which can save time and resources that can then be used in more important and creative tasks.(5)

University professors’ knowledge of AI is fundamental to develop skills and competences in students, to improve the quality and efficiency of learning, high-quality scientific research, and to optimize administrative and managerial tasks. Therefore, it is necessary that university professors get trained and updated in their knowledge of emerging technologies, like AI, in order to adapt and contribute to the development and evolution of the educational sector in the digital age.

The goal of this study was to determine the sustainable factorial structure of the dimension of relations of an instrument to assess the level of knowledge of artificial intelligence in university professors and to determine the discriminating validity of the sustainable factorial model and the reliability of the instrument obtained in the sustainable factorial model.   

 

METHODS

Type of study: a cross-sectional, metric validating study was carried out for the purpose of developing and validating an instrument to assess the level of knowledge of artificial intelligence in university professors.

Techniques and procedures: a sample of 83 university professors was chosen. An instrument on artificial intelligence for university professors was applied, said instrument being composed of 15 questions divided into three sections. A psychometric analysis was conducted in order to assess its validity and reliability.

Psychometric analysis: the fit indicators of the structural model of the instrument were assessed, such as the χ2/gl ratio, Joreskog goodness-of-fit index (GFI), Bentler-Bonett normed fit index (NFI), Bentler comparative fit index (CFI) and the root mean square error of approximation (RMSEA). A reliability analysis was carried out by calculating the internal consistency of each subscale and the instrument as a whole, using Cronbach's alpha coefficient. The corrected item-subscale correlations were analyzed in order to assess the internal consistency of each subscale and section. 

Statistical processing: the data were analyzed with the statistical software SPSS version 25.

Ethical aspects: informed consent of the participant university professors was obtained, who were informed of the nature of the study, its scope and the purposes of the instrument. The collected data were used only for academic purposes and the anonymity of the participants was guaranteed.

 

RESULTS AND DISCUSSION

The proposed instrument consisted of 15 questions, 5 for each of the three constituent parts:

·           Part 1 – Theoretical aspects of Artificial Intelligence

·           Part 2 – Artificial Intelligence Tools for Education

·           Part 3 – Uses of Artificial Intelligence Tools in the Classroom and in Teaching Activities

 

Reliability

So as to assess the psychometric properties, a confirmatory factorial analysis (CFA) and a reliability analysis were carried out.

CFA assesses whether the structure of the instrument (the questions and the answer options) adequately shows the constructs of interest (theoretical aspects of AI, AI tools for education and uses of AI tools in the classroom and in teaching activities).(6)

In the sample of university professors that was analyzed, the following results were obtained:  

·            The average of correct answers in Part 1 (theoretical aspects of AI) was 3,8 with a standard deviation of 0,9;

·            The average of correct answers in Part 2 (AI tools for education) was 3,5 with a standard deviation of 1,2;

·            The average of correct answers in Part 3 (uses of AI tools in the classroom and in teaching activities) was 4,2 with a standard deviation of 0,8.

The results of the analysis show that the model adequately fits the data, which suggests that the instrument adequately measures the constructs of interest.

In order to assess the reliability of the proposed instrument, Cronbach's alpha coefficient was calculated for each Part. The results show that the alpha coefficient of Part 1 is 0,77, the alpha coefficient of Part 2 is 0,65 and the alpha coefficient of Part 3 is 0,83. These values indicate that the instrument has good internal consistency and can be used reliably to assess the level of knowledge of artificial intelligence in university professors.

As partial results, we can affirm that the confirmatory factorial analysis and the reliability analysis suggest that the instrument has good psychometric properties and is valid and reliable to assess the level of knowledge of artificial intelligence in university professors.

 

Validity of the construct

Table 1 shows the descriptive statistics of the elements of the questionnaire (corrected item-subscale correlations, alpha coefficients, averages and typical deviations).

 

Table 1. Descriptive statistics of the elements of the questionnaire

Part

Ask

Corrected item-sub-scale correlation

Coefficient alpha

Mean

Standard deviation

1

1

0,63

0,77

3,8

0,9

1

2

0,67

1

3

0,58

1

4

0,71

1

5

0,52

2

6

0,45

0,65

3,5

1,2

2

7

0,58

2

8

0,71

2

9

0,63

2

10

0,48

3

11

0,62

0,83

4,2

0,8

3

12

0,57

3

13

0,69

3

14

0,76

3

15

0,63

 

As can be seen, the table shows the corrected item-subscale correlation, the alpha coefficient, the average and the typical deviation of each question in relation to its respective subscale. The alpha coefficients for each subscale (0,77 for Part 1, 0,65 for Part 2 and 0,83 for Part 3) indicate that the instrument has good internal consistency and the questions within each subscale are related to one another. Besides, the averages and typical deviations of each subscale suggest that university professors’ level of knowledge is slightly higher in Part 3, related to the uses of AI tools in the classroom and in teaching activities.

Subsequent to this analytical process, a confirmatory analysis of the model generated in the exploratory study was carried out by using the method of structural equations. By means of this new analysis, the fitness of the model was confirmed, since a sustainable model made up of the identified factors and the total indicators (table 2) was obtained. 

 

Table 2. Structural model of the dimension of relations of the instrument

Part

Ask

Load factor

Variance explained

1

1

0,78

0,61

1

2

0,79

0,63

1

3

0,71

0,50

1

4

0,82

0,67

1

5

0,63

0,40

2

6

0,56

0,31

2

7

0,69

0,47

2

8

0,83

0,69

2

9

0,66

0,44

2

10

0,56

0,31

3

11

0,76

0,58

3

12

0,68

0,46

3

13

0,81

0,66

3

14

0,87

0,76

3

15

0,76

0,58

 

The table shows the factor load, ie the relation between each question and its respective subscale, and the explained variance, which indicates the amount of variance in the subscale that is explained by each question.

The factor load values are significant to all questions, which indicates that each question is strongly related to its respective subscale. Furthermore, the explained variance values are high for all questions, which indicates that the questions are representative of the subscale and are coherently related to the constructs of interest.

When applying the chi-square, it indicates that there are no significant differences in the explanatory power between the saturated model and the proposed model (table 3).

 

Table 3. Fit indicators of the structural model of the instrument

Indicator

Value

Ratio χ2/gl

2,1

Joreskog Goodness of Fit Index (GFI)

0,93

Bentler-Bonett Normed Fit Index (NFI)

0,89

Bentler Comparative Fit Index (CFI)

0,91

Root Mean Square Error of Approximation (RMSEA)

0,07

 

The fit indicators of the structural model suggest that the model adequately fits the data. The χ2/gl ratio of 2.1 indicates good fit of the model, and the GFI, NFI and CFI values are close to 1, which indicates good fit of the model. Moreover, the RMSEA value of 0,07 indicates moderate fit of the model, which suggests that the model is adequate to assess the level of knowledge on artificial intelligence in university professors.

 

CONCLUSIONS

The results of the confirmatory factorial analysis demonstrated that the instrument on artificial intelligence for university professors has a sustainable and coherent factorial structure, with three subscales representing the theoretical aspects, the educational tools and the uses of AI in the classroom. This indicates that it can become an adequate tool to assess university professors’ level of knowledge of AI and the fact that the questions are coherently related to constructs of interest.

The factor load values for each question and subscale indicate that the questions are strongly related to their respective subscale and it effectively measures the constructs of interest. Besides, the explained variance values are high, which indicates that the questions are both representative of the subscale and coherently related to the constructs of interest. These results support the validity of content and discriminant of the sustainable factorial model.

The results of the reliability analysis indicate that it has high internal consistency, with Cronbach's alpha coefficient values higher than 0.8 for each subscale and globally. Moreover, the corrected item-subscale correlations are high, which indicates that the questions consistently measure the constructs of interest. These results support the reliability of the instrument obtained in the sustainable factorial model.

The results of this study support the validity, reliability and sustainable factorial structure of the instrument on artificial intelligence for university professors, which make it an adequate tool to assess university professors’ level of knowledge of AI.

Those university professors having knowledge of AI could be better prepared to integrate these emerging technologies into their teaching strategies and improve the quality and efficiency of learning. Therefore, we recommend that training and updating programs for university professors in the area of AI should be implemented in order to improve the quality of learning and the training of the future professionals.

 

REFERENCES

1.    Ocaña-Fernández Y, Valenzuela-Fernández LA, Garro-Aburto LL. Inteligencia artificial y sus implicaciones en la educación superior. Propósitos y Representaciones 2019;7:536-68. https://doi.org/10.20511/pyr2019.v7n2.274.

 

2.    Moreno Padilla RD. La llegada de la inteligencia artificial a la educación. Revista de Investigación en Tecnologías de la Información: RITI 2019;7:260-70.

 

3.    Bueno de Mata F. Macrodatos, inteligencia artificial y proceso: luces y sombras. Revista General de Derecho Procesal 2020:3.

 

4.    López MT. Tendencias e impacto de la inteligencia artificial en comunicación: cobotización, gig economy, co-creación y gobernanza. Fonseca, Journal of Communication 2021:5-22. https://doi.org/10.14201/fjc-v22-25766.

 

5.    Valbuena R. Inteligencia Artificial: Investigación Científica Avanzada Centrada en Datos. Cencal Press; 2021.

 

6.    Brown TA. Confirmatory Factor Analysis for Applied Research, Second Edition. Guilford Publications; 2015.

 

FINANCING

No external financing.

 

CONFLICT OF INTEREST

No conflicts of interest.

 

AUTHORSHIP CONTRIBUTION

Conceptualization: Camilo Andrés Silva-Sánchez.

Research: Camilo Andrés Silva-Sánchez.

Methodology: Camilo Andrés Silva-Sánchez.

Original writing-drafting: Camilo Andrés Silva-Sánchez.

Writing-revision and editing: Camilo Andrés Silva-Sánchez.

 

 

 

SUPPLEMENTARY MATERIAL 1

Part 1 - Theoretical Aspects of Artificial Intelligence

1. How would you define artificil intelligence?

a) A branch of computer science focused on creating intelligent machines that can learn from experience and carry out tasks normally requiring human intelligence.

b) A type of technology using robots to carry out automated tasks in factories and other industries.

c) A programming technique using high level languages to develop complex software applications.

2. What is the difference between supervised learning and unsupervised learning in artificial intelligence?

a) Supervised learning entails the classification of data into predefined categories, whereas unsupervised learning entails the identification of patterns in the data without any predefined categories.

b) Supervised learning is a process in which the artificial intelligence algorithms learn from labeled training data, whereas unsupervised learning does not use any labeled data to learn.

c) Supervised learning is used to classify images and detect fraud, whereas unsupervised learning is used to segmet data and detect anomalies.

3. What are neural networks and how are they used in artificial intelligence?

a) Neural networks are a set of automatic learning algorithms inspired on the structure and functioning of the human brain and they are used to classify and predict data.

b) Neural networks are a set of programming techniques used to develop game and 3-D animation software.

c) Neural networks are a data mining technique used to identify patterns in big datasets.

4. What is reinforcement learning and how is it used in artificial intelligence?

a) Reinforcement learning is an automatic learning technique using a reward and punishment system to teach a machine how to carry out specific tasks.

b) Reinforcement learning is a programming process used to develop word-processing software.

c) Reinforcement learning is a data mining technique used to identify patterns in big datasets.

5. What are some of the ethical and social challenges related to artificial intelligence?

a) Algorithmic biases, privacy, social and ethical responsibility.

b) Compute power, safety, efficiency.

c) Innovation, processing speed, accessibility.

 

Part 2 - Artificial Intelligence Tools for Education

1. What is natural language processing and how can it be used in education?

a) Natural language processing is an artificial intelligence technique used to analyze and understand human language and it can be used in education to develop virtual learning assistants.

b) Natural language processing is a data mining technique used to identify patterns in big educational datasets.

c) Natural language processing is a programming technique used to develop teaching application software.

2. What are some of the artificial intelligence tools that can be used in education?

a) Virtual learning assistants, recommender systems, educational chatbots.

b) Video editing software, content management platforms, search engines.

c) Graphic designing software, text editors, spreadsheets.

3. What is learning analytics and how can it be used in education?

a) Learning analytics is an artificial intelligence technique used to analyze and understand the learning patterns of students and it can be used in education to improve teaching and learning.

b) Learning analytics is a programming technique used to develop teaching application software.

c) Learning analytics is a data mining technique used to identify patterns in big educational datasets.

4. What are educational chatbots and how can they be used in education?

a) Educational chatbots are artificial intelligence programs used to interact with students and provide answers to their questions.

b) Educational chatbots are a programming technique used to develop teaching application software.

c) Educational chatbots are a data mining technique used to identify patterns in big educational datasets.

5. What are some of the advantages of using artificial intelligence tools in education?

a) Customized learning, instant feedback, efficiency in time management.

b) More interaction among students, reduced costs, more access to information.

c) Improvement in the quality of learning, development of technological skills, more motivation of students.

 

Part 3 – Uses of Artificial Intelligence Tools in the Classroom and in Teaching Activities

1. How can artificial intelligence be used to improve teaching in the classroom?

a) Using data analysis tools to customize students’ learning.

b) Developing educational chatbots to interact with students and provide instant feedback.

c) Creating simulations of educational situations to improve practical learning.

2. How can artificial intelligence be used to improve the evaluation of students?

a) Using data analysis tools to evaluate students’ performance and provide customized feedback.

b) Developing automatic evaluation systems using automatic learning techniques to automatically grade students’ answers.

c) Creating evaluation environments based on games that use artificial intelligence techniques to evaluate students’ performance.

3. How can artificial intelligence be used to promote collaboration and teamwork in the classroom?

a) Developing data analysis tools to evaluate students’ performance in teams and provide customized feedback.

b) Creating team chatbots helping to coordinate and communicate with students in team projects.

c) Developing simulations of teamwork situations allowing students to practice and improve their collaborative skills.

4. How can artificial intelligence be used to develop technological skills in students?

a) Creating educational projects that use artificial intelligence tools to solve problems of the real world.

b) Developing simulations and educational games that teach concepts of artificial intelligence and programming.

c) Creating extracurricular programs that allow students to learn about artificial intelligence and programming.

5. What ethical and social challenges should be taken into account when using artificial intelligence in education?

a) Privacy and protection of data from students, justice and equity in education, social and ethical responsibility when developing artificial intelligence tools.

b) Technological complexity, lack of resources and training of professors, interoperability among artificial intelligence tools. 

c) Accessibility, sustainability and cost of the implementation of artificial intelligence tools in education.