Knowledge Transfer through Mobile Application Needs Questionnaire: Validity and Reliability

Introduction: Knowledge transfer through mobile application needs assessment to develop a userfriendly technology. However, the means to measure the need is unavailable. An instrument was developed from the Joanna Briggs new model of EBP to examine the nurses’ needs. This study aims to examine the reliability and validity of knowledge transfer through mobile application needs questionnaire for nurses. Methods: The validity of the questionnaire was examined for its content with Content Validity Index (CVI). CVI was measured for its relevance, ambiguity, clarity and simplicity on each item (I-CVI) and scale (S-CVI). Content validity assessment forms and guidelines were given to six experts in nursing and computer sciences. Their suggestions contributed to the questionnaire revision. Subsequently, the online questionnaire was completed by 40 nurses randomly selected in a hospital in Depok City. Internal consistency was analyzed for the reliability. Results: The CVI of this questionnaire is high with mean of I-CVI and S-CVI at 0.96 relevance, 0.98 ambiguity, 0.97 clarity and 0.94 simplicity. Cronbach’s alpha shows that the internal consistency is acceptable (0.87). Conclusion: The questionnaire is content valid and reliable. This questionnaire can be used to measure those needs and contribute to the design of a user-friendly mobile application.


INTRODUCTION
Barriers hinder evidence-based practice (EBP) in South East Asia, its lack being partly rooted in inadequate access to quality healthcare information (McDonald et al., 2010). Indonesian nurses, doctors, and midwives were amongst the participants investigated by the South East Asia Optimizing Reproductive and Child Health in Developing Countries (SEA-ORCHID) study. Some of the barriers noted in the projects were time constraints, lack of capacity in discovering, evaluating and interpreting evidence, and inadequate access to evidence (McDonald et al., 2010;Turner, 2009). Adding to that, nurses in a hospital in Padang reported research articles were not readily available (Novrianda, Hermalinda, & Abdullah, 2018).
However, access and readiness to healthcare information through ICT are positive in Depok City (Fajarini, Rahayu, & Setiawan, 2020). Internet, computers or laptops are available in all healthcare providers for doctors and nurses. These reports are in conjunction with surveys reporting high internet penetration and usage to seek health information in urban regions (Asosiasi Penyelenggara Jasa Internet Indonesia, 2018;CIGI-Ipsos, 2017). Doctors and nurses expressed interest and need to look for evidence, although, only nurses reported relationship between evidence-based practice and ICT. However, the relationship was weak and negative. This study does not corroborate with the United Nations (2015) and WHO-SEARO (2016), who suggested ICT to enhance EBP. Active dissemination as a part of evidence transfer (Jordan, Lockwood, Munn, & Aromataris, 2019) is necessary. Specifically, Doran et al. (2010) recommended mobile information technologies to overcome EBP barriers and the WHO (2011) has established mobile Health (mHealth) for information initiatives. As well, ICT has been made available and accessible by the stakeholders for nurses in Depok city to improve their EBP . Therefore, the need for knowledge transfer through ICT is required to provide data for the development of nurse-friendly knowledge transfer mobile application. Some studies reported ICT extensive usage (Arthur, Kable, & Levett-Jones, 2011;Button, Harrington, & Belan, 2014;Miller, Graves, Jones & Sievert, 2010) and analysis among nurses (Christiansen, Fagerström, & Nilsson, 2017;Clarke et al., 2013;Fagerström, Tuvesson, Axelsson, & Nilsson, 2017;Gerrish et al., 2006;Ward, Stevens, Brentnall, & Briddon, 2008), However, instruments measuring nurses' needs of ICT for EBP are not available. Needs analysis is a prerequisite stage in developing mobile application (Potnis, Regenstreif-Harms & Cortez, 2016;Sommerville, 2016). Thus, a questionnaire was developed to obtain data of the needs, but its validity and reliability must be tested prior to the survey. This study is conducted to examine its reliability and validity.

Design
A descriptive design was assigned in this study. The content validity test was examined by experts and the reliability test was conducted to nurses. The experts were purposively recruited from clinical and academic areas in nursing and computer sciences based on their expertise. The experts comprised of three registered nurses from two hospitals, a lecturer in nursing and two lecturers in computer sciences. All nurses have over 10 years of experience as clinicians and managers. One nurse has a master degree in nursing and two are undergoing master degrees in nursing. The lecturer in nursing has 20 years of experience as a lecturer and is currently undergoing a doctoral program majoring in nursing management. Experts in computer science have a master degree, one expert in mastering mobile applications and one in big data. For the content validity test, a minimum of three and no more than 10 experts are suggested to examine an instrument and to reach its validity of the content (Lynn, 1986), thus the number of experts in this study (six) have met the requirement. A survey was assigned to evaluate the reliability test of this instrument. A total of 40 randomly selected nurses participated in the reliability test.

Settings
These reliability measurements were conducted at a hospital in Depok City in August 2019. The hospital was chosen based on its capacity in EBP judged from previous study (Fajarini et al., 2020;Setiawan et al., 2018).

Questionnaire Development and Testing
The questionnaire was developed from the Joanna Briggs Institute new model of evidence-based practice, the evidence synthesis and transfer from the pebble of knowledge (Joanna Briggs Institute, 2017). Evidence synthesis comprises types of evidence, which are evidence summary, systematic review, and clinical guidelines.
Evidence transfer emphasizes the delivery of the evidence to be comprehensible and applicable, to meet the information needs and to be cost-effective (Pearson, Weeks, & Stern, 2011). The transfer involves active dissemination, education programs, and system integration.
Interesting formats, recommendations, knowledge leaders and information technology are mentioned as methods of active knowledge transfer. Therefore, this questionnaire contains two scales, which are the evidence types and methods of their delivery.
The questionnaire comprises three sets of questions in the evidence type scale and 13 questions on the methods scale. In each evidence type, the nurses were asked how important were the evidence types in the success of their performance and how well they were able to understand it using a 7-pointLikert scale (table 1). In the methods scale, 12 features of the mobile application were enquired of their importance using a 5point Likert scale and one set of questions asking how important is knowledge transfer through a mobile application and how well the nurses were able to comprehend knowledge through a mobile application.

Item Analysis
Each set of data in the evidence type and method scales was then interpreted into the need of the nurses. The data were categorized into four quadrants. The data were considered within the quadrant of need with high priority when they were rated high as how important and how well items. Where the items rated high on how important but low on how well, they were considered needed with low priority. Items not needed rated low in how well it will be understood and fell into the two other quadrants.

Validity: Content Validity
Content validity is commonly used in nursing education to see whether an instrument has represented the existing concept (Heale & Twycross, 2015). It can be done by asking the experts to rate each question and obtain an opinion. The Content Validity Index (CVI) of the questionnaire was determined by four elements. Lynn (1986) noted relevance as the only element whilst Yaghmale (2009) added three more elements, which are simplicity, clarity, and ambiguity on a 4-point Likert scale. The relevance scores one as irrelevant, two as somewhat relevant with major revision, three as relevant with minor revision, and four as relevant. The simplicity, clarity and ambiguity mark one as a total revision of item is required, two as major revisions, three as minor revisions and four as good. The examiners were given content validity assessment forms and guidelines on how to assess the content validity of the study via e-mails. Scores were compiled and accounted from all experts. Suggestions given by experts were taken into consideration for the improvement of the questionnaire. CVI was measured from the proportion of each item (I-CVI) and all items, known as Scale CVI (S-CVI) (Lynn, 1986). I-CVI was proportioned for each item by summing the number of experts scoring 3 and 4 of the item and dividing it with the total number of experts (6). Mean I-CVI was calculated from the total proportion of all items and divided by total items (16). S-CVI average (S-CVI/Ave) followed similar rules as I-CVI. The number of items which scored 3 and 4 from each expert was added up and then divided by 16. S-CVI/Ave was measured from the total proportion of all experts and then divided by the total number of experts. S-CVI Universal Agreement (S-CVI/UA) was measured from the number of valid items divided by the total items (16). CVI reports have been criticized, but the critiques recommend reports to comply with Lynn's criteria (Polit & Beck, 2006). For six experts, an item is valid if the value scored no less than 0.78. Thus, items valued 0.78 and beyond remain in the questionnaire. Several studies have exercise CVI test (Sh, Hsu, Toobert., & Wang, 2019) with three to five judges (Emmanuel & Clow, 2017;.

Reliability: Internal consistency reliability
Reliability is used to see whether each respondent's answer has consistency (Heale & Twycross, 2015). Cronbach's alpha is frequently applied as a reliability test (Cronbach, 1951). The data collected were analyzed using its formula and Cronbach's alpha 0.7 and above shows the internal consistency of the questionnaire as acceptable, good, and excellent (Heale & Twycross, 2015). The reliability test was conducted at several wards in a hospital which participated in this study. Explanation about the study and terms in the questionnaire were given prior the test. The nurses completed the online questionnaire in the presence of the researchers.

Ethical Approval
Ethics of this study were approved by the Faculty of Nursing, Universitas Indonesia with number SK-178/UN2.F12.D1.2.1/ETIK.FIK.2019. This study was explained to the participants and informed consent was obtained. Privacy was guaranteed with the participants' anonymity.

Content Validity
The CVI was calculated based on relevance, ambiguity, clarity, and simplicity. Table 1 shows I-CVI range from 0.67 to 1.00. The mean of the I-CVI result is 0.96 for relevance, 0.98 for ambiguity, 0.97 for clarity, and 0.94 for simplicity as well as the mean of S-CVI. Whereas, the S-CVI/UA is 12 items out of 16 (0.75), which implies 75% of the items are relevant. Most were judged valid for ambiguity (88%), clarity (81%), and simplicity (69%). All S-CVI/Ave are above 0.90.

Reliability
Internal consistency results obtained a Cronbach's alpha value of 0.87. It means that this questionnaire is reliable to be used in the true population. The sample of this reliability test is 40 nurses at one hospital; future research needs to conduct a reliability test with larger or various sample.

DISCUSSIONS
The relevance, ambiguity, clarity, and simplicity of this questionnaire are high at item and scale level. Over 90% of the items were considered relevant by almost all experts. It implies that this questionnaire covers the content of items accordingly. Most of the items had I-CVIs of 1.00, and the lowest I-CVI was 0.67 only at one item. For ambiguity, the result of S-CVI/UA was higher than the others, while for simplicity it was the lowest. This reflects that the meaning of the items is ascertained but less simple. Similarly, several low values were noted by Emmanuel and Clow's (2017) instrument, but remained in the questionnaire and changes were made based on the participants' feedback. Fortunately, experts in this study suggested changes.
Experts suggested several improvements in the content and sentences. Revision of instruments based on suggestions is possible (Yaghmale, 2009). They criticized the use of evidence summary and systematic review because these terms are unfamiliar to the majority of nurses. Therefore, the researchers provided explanations and samples of both forms of evidence as they collected the data. "Best evidence recommendations" were changed from recommendation for clinical decision because clinical decision based on evidence is too complex to collect using a questionnaire alone. Thus it was focused on the best evidence. "Interactive discussion forum" was changed from communication with experts as it is easier to understand. Examples were added at the search using keywords and catalogs items to describe its intended meaning. Lynn (1986) also suggests items less than the requirement to be eliminated or revised. Item evidence inquiry was judged low for simplicity (0.67) because it was thought as duplication with searching using keywords and catalog items. Thus, it was revised by giving an explanation of what it means to avoid perception of duplication and changed to "Evidence inquiry (you can submit a request of evidence you need)". The questionnaire was revised based on these suggestions prior to the data collection.
This questionnaire's content validity is assured. Polit and Beck (2006) recommended CVI to meet the minimum agreement as being set by Lynn (1986) for excellent content validity at 0.78 for six experts and 1.00 for less than six experts. Also, with SCVI/Ave higher than 0.90. This questionnaire has met the requirements. Higher values were found in Rahayu et al. (2018) and Sugiharto et al.'s (2019) questionnaires with mean I-CVI 0.99 and 1.00, respectively. However, Sugiharto et al.'s (2019) study did not mention the number of experts involved, while, there were three experts in Rahayu et al.'s (2018) examination.
The reliability of this questionnaire is over 0.8, suggesting it is good for data collection. It is internally consistent. Overall, this instrument has exceeded the minimum requirement of validity and reliability; therefore, all items remain in the questionnaire and the content is considered valid and reliable.
The requirement of knowledge transfer of nurses needs through mobile application needs to be explored to obtain data on its content and features. This questionnaire can be used to measure those needs and contribute to the design of a mobile application that is userfriendly. The limitation of this study is that the reliability test was conducted at a type C private hospital. Thus, it should be taken into consideration when exercising this questionnaire to nurses from other types of hospital.

CONCLUSION
This knowledge transfer through mobile application needs questionnaire was tested and shown to have excellent validity and reliability. This questionnaire is required to provide data for the development of user-friendly knowledge transfer mobile application. Future study needs to conduct a validity and reliability test with larger or various sample. Further examination at various healthcare settings is recommended.