From Researchwiki

• Print Version
• Media:ASCILITE2007.odt (Image:ASCILITE2007.odt)
• Media:ASCILITE2007.pdf (Image:ASCILITE2007.pdf)
• Media:ASCILITE2007.doc (Image:ASCILITE2007.doc)

Progress

Notes

Due: 23 July 2007.

What are the big features/findings of the thesis to illustrate?

Full papers are limited to 10 pages and 2 Mb

Two topics areas we can fit into are:

1. Social networking and web 2.0
2. Online role play

Suggested title:

• Peacock, T, Fellows, G & Eustace,K. (2007). The quality and trust of wiki content in a learning community

The theme for ascilite 2007, "ICT: Providing choices for learners and learning", focuses on catering for the diversity of learners and learning, and how we as educators can provide stimulating and engaging learning environments and experiences for all our learners through the use of ICT in higher education.

My quick start below shows 6 main headings with no. of pages devoted in [ ] as approx. guide.

1. Abstract [.1]
2. Introduction [.2]
3. Problems with wikis [1.1]
4. Research questions [ .6]
5. Methodology and experimental design [2]
6. Results and Findings [5]
7. Concluding comments [1]

Submission and review is done online at:

• http://www.ascilite.org.au/conferences/singapore07/review/index.php?authorsInstructions=1

Similar paper but Trevor's thesis is much deeper project:

• Implementing wiki software for supplementing online learning

Choy, S. O. and Ng, K. C. (2007). Implementing wiki software for supplementing online learning. Australasian Journal of Educational Technology, 23(2), 209-226. http://www.ascilite.org.au/ajet/ajet23/choy.html

Refs style for when writing the paper:

• Beasley, R.E. & Vila, J.A. (1992). The identification of navigation patterns in a multimedia environment: A case study. Journal of Educational Multimedia and Hypermedia, 1(2), 209–222.
• Chen, C. J. (2006). The design, development and evaluation of a virtual reality based learning environment. Australasian Journal of Educational Technology, 22(1), 39-63. http://www.ascilite.org.au/ajet/ajet22/chen.html
• Kearsley, G. (2004). Explorations in learning & instruction: The theory into practice database http://tip.psychology.org/ [viewed 8 May 2007].
• O'Shea, T. & Self, J.A. (1983). Learning and teaching with computers. Englewood Cliffs, NJ: Prentice-Hall Inc.
• Underwood, J. (1997). Breaking the cycle of ignorance: Information technology and the professional development of teachers. In D. Passey & B. Samways (Eds.), Information Technology: Supporting change through teacher education. (pp.155-158). London: Chapman & Hall.

Links

• http://www.ascilite.org.au/conferences/singapore07/
• http://www.ascilite.org.au/ajet/ajet23/res/choy.html
• http://www.ascilite.org.au/ajet/ajet22/chen.html
• http://www.ascilite.org.au/conferences/singapore07/papers/papers.htm

Paper

ASCILITE_2007/Paper

The quality and trust of wiki content in a learning community

Trevor Peacock, Geoff Fellows, G & Ken Eustace
Charles Sturt University

Abstract

User generated content is having an ever-increasing influence and presence on the Internet. Wiki communities, in particular Wikipedia, have gained wide spread attention and criticism. This research explores criticisms and strengths of wiki communities, and methods to reconcile the two. This research tests wiki software in an educational setting to determine indicators of article quality. The results give insight into the use of wiki systems in educational settings, suggest possible methods of improving the validity of content created within wiki communities, and provide groundwork for further research in the area.

Introduction

The Internet has seen the astonishing growth of blogging, RSS, and podcasting, as forms of user-generated content. Blogs are replacing traditional news sites, and online discussion and interaction, through the popularity of sites such as digg and Slashdot are changing the way we find, judge and trust information. Wikis have continued this trend in user-built interactive information universes. Wikipedia, a free, open-content encyclopaedia has popularised the concept of a wiki, with many projects adopting MediaWiki, the software used by Wikimedia for Wikipedia, or creating their own custom wiki systems.

Leuf and Cunningham (2001), p. 14 (cited in Schwall 2003) define a wiki from a technological standpoint as

From a conceptual view the Wikipedia Contributors (2006a) describe a wiki as:

Wikis are gaining widespread use in business, in an open online format, and in education. This research sought to understand the problems with the reliability and trust of content in wikis, and how these problems can be solved.

Problems with wikis

Open source and wikis break the traditional model of publishing; rather than authors publishing in their own spaces, and competing for an audience, authors contribute to the same space, attempting to improve the collective writing of the community. To paraphrase Graham (2005); people contribute what they like, the good stuff stays, the bad gets removed. Herein lies the major criticism of wikis in general, that quality can not "evolve" from this process. There is no guarantee of the accuracy of content, and there is no formal process of validation, by which content is said to be correct. Rather, a continual process is used, where content is constantly being validated and edited, and accuracy is transitory.

The problems with wikis, can be demonstrated by the criticism of Wikipedia. Wikipedia's criticism extends primarily from its promotion as an encyclopaedia. Using this label "carries a powerful connotation of reliability" (Orlowski 2005), something which the Wikipedia community can not guarantee. Building an encyclopaedia is an admirable and inspiring goal that drives contributors, leading to Wikipedia's success today.

The principal criticism of Wikipedia as an encyclopaedia is that there are no limits on who may edit content, and the lack of a formal peer review process. The freedom of editability is one of Wikipedia's greatest advantages, and has been the factor that has allowed Wikipedia to grow at such a phenomenal speed. Nupedia employed a strict editorial process of peer review that ultimately brought development to a crawl. Wikipedia abandoned such limitations completely. This open editing is one of the most important philosophies of the Wikipedia community, allowing people to contribute anonymously, and refine/fix other contributors work.

Critics say that allowing "any fool" to edit the encyclopaedia is a great detriment to the encyclopaedia, allowing poor quality content to enter into the encyclopaedia. The community however disagrees; poor work is removed or repaired by the community.

Research questions

This research focused to determine an answer to one question. The question asked was as follows.

One proposed method is to validate articles by an "expert" (individual or group). To make this process easier, good articles may be identified by the community, to be passed to the expert to validate, thus eliminating poor articles from ever beginning, and ultimately failing the validation process. This research focused on two such mechanisms, user ratings, and attention data. The following general questions were investigated by this research, using the methods presented in this chapter:

• Can ratings made by users be reliably used to identify quality articles?
• Can attention data be used in an wiki environment to estimate the quality of an article?

In order to answer these questions, this research uses a range of research methodologies to process different types of data, and to triangulate results from these different sources to provide a clearer picture (Preece 2000). Considering the little research done in the field, this research looks at a wiki in a naturalistic setting, focusing on aspects of quality arising from a real-world wiki community.

The community studied was one created for this research, formed from the students of a "computer supported collaborative work" subject at CSU. Using this group of participants also allowed observations to be made of their participation in an authentic learning setting allowing the effectiveness of wikis in such settings?

Methodology and experimental design

This research employed two formal data collection methods. This research implemented a wiki, and designed a rating system, which was trialled in a naturalistic setting while log data was gathered. Building from the results of this wiki experiment, a survey was conducted. The server logs allowed participants behaviour to be analysed, while the survey aided in interpreting results from the server logs, and helped to answer any remaining questions. Observations from the wiki, and the development process were also recorded.

A wiki community was established and monitored, by implementing a wiki, extended with a simple page rating mechanism. The rating mechanism allowed users to rate an article within the wiki, using a standard 5-star rating. A group of undergraduate students partaking in an undergraduate IT subject were invited to the wiki.

MediaWiki 1.7.1 was the version implemented (the most up to date stable version at the time of installation), it was configured for public access, and seeded with a set of pages explaining the purpose of the wiki, instructions for editing the wiki, and the set of assignment tasks to be completed by the students.

The seed pages provided communal areas for communication and finding information relevant to the students' tasks. Certain key pages were omitted, with the intention that students should create these based on instructions provided. Example pages were also added, as a guide to students when writing their own pages.

Over the course of the experiment, the wiki was monitored, and kept tidy by the researcher (who filled the "janitorial" role), following the guidelines set out by the Wikipedia Contributors (2006b).

Participants in the wiki were students from the ITC213 undergraduate class for spring 2006. ITC213, or Computer Supported Collaborative Work teaches a range of social and technical topics surrounding online communities. The class was volunteered by the lecturer and coordinator Ken Eustace. The subject is a very hands on subject where students collaborate using various collaborative tools (Charles Sturt University 2006).

Students in the class are placed into Pools of Online Dialogue (POD) groups, and in those groups, complete fortnightly collaborative exercises (Eustace 2006). This research was designed to provide activities for the first two POD exercises.

The POD exercises were designed with three goals in mind:

• Provide a suitable exercise for learning and assessment
• Generate valuable content for the wiki
• Promote typical wiki behaviour

A wiki does not generally have a set of tasks that each user completes, rather there may be a to-do list that volunteers may complete, with no requirement to do so. For the participants of this research however, it was required that there be a compulsory element to the activities to ensure participation, and to facilitate the academic assessment of the students work.

The set exercises attempted to set, where possible, few limitations regarding the type of interaction within the wiki. This was to try to accurately simulate a wiki community, where there are generally no such limitations. Students were required to contribute 500 words for each of the two activities, however the distribution of these 500 words was not limited. The words could be "spent" adding to existing articles, or pooled with a group of people to generate a larger article.

Students were also encouraged to rate other articles they read, and to be creative, letting ideas flow, making students search, rate, contribute to, and learn from each others content. These activities were designed to encourage an organic (Cunningham 2006, Ma 2005, Wagner 2004) site, where students may otherwise not have been used to studying this way.

Once data was collected, articles were analysed to determine an objective measure of quality. These measurements provided baseline measure of the quality of articles, for comparison with system generated ratings.

After work in the wiki had completed, a follow-up survey was employed, where participants were asked to answer a questionnaire, comprising of questions arising from observations and results from the wiki experiment phase. This data was used to aid in the interpretation of the results from the wiki logs, and was aimed to determine users' perceptions, opinions, feelings and motives regarding their participation in the wiki.

In summary, five data sources were used. These were article content, the content created by users of the wiki, including any textual content, discussions, comments, and historic revisions of articles; ratings, as determined by users of the wiki through the embedded rating system; server logs, allowing user behaviours to be monitored, to see how users interact with the system, and to determine time spent using the system, including attention data; survey results, general demographics, users' computer self-efficacy, and user thoughts on the use of the wiki and rating system; observations from the live wiki community.

Results and Findings

With no set due date for completion of POD exercises, contributions to the wiki were made in fairly regular intervals. It was observed that most students would post an entire article with a single edit. It seemed that students preferred to draft their writing in an off-line system (perhaps a word processor). There were however a few exceptions, where students may post a paragraph at a time. There was little modification to text however, after its original submission.

The exception to this contribution style were the POD group pages, pages students were specifically asked to use to collaborate between members.

These pages were not created when seeding the wiki, they were left for students to create. Much editing of these pages was observed, usually by several members of the group. These pages were frequently updated with often minor presentation changes, following each others leads, and fixing each others mistakes. It seemed these pages were perceived common property, whereas where a single student posted their writing on a page of its own, students would not venture to interfere.

Two formal data sources were used for analysis in this research. Firstly Apache logs collected from the wiki community, and secondly data from a follow up survey of the participants.

Apache Logs

The wiki was installed and initialised on the Internet Special Projects Group (ISPG) server on the 4/Jul/2006. The ISPG server, maintained by Geoff Fellows, hosts several applications for research projects, as well as teaching resources. Over the following weeks, the wiki was seeded with its initial set of pages. Data recovered from ISPG contained 15162 lines of logs pertaining to KakapoWiki, recorded between 31/Jul/2006 and 20/Sep/2006. From this number were removed requests for files (3234) (not to the MediaWiki engine itself), including CSS files and JavaScript files requested through the engine (2119 and 762 respectively). 2436 lines generated from automated (non-human) requests to the wiki were also ignored. 5 lines also failed to be matched by the regular expression being used to parse log file records. None of these five pertained to the wiki. In total 6606 usable entries (table 4.1) were retrieved from the Apache logs (non-automated queries to the engine). These entries show users actions or behaviours while interacting with the wiki.

One goal of this experiment was to monitor a simple rating system in a wiki community, and users behaviours and reactions to the system, where the participants were undergraduate students. Apache logs provide a rich source of objective measurements of behaviour, however they do not reveal intent or the thoughts and feelings behind the behaviour.

Some of the 81 recorded ratings were filtered from the results to improve reliability. The filtered ratings were ratings where (i) the user rated a page a second time (i.e., corrected a rating, or re-assessed a page), (ii) the user rated a page they themselves had recently edited. Table 1 summarises ratings based on two determining factors.

In total, 31 usable ratings were generated. Comparing ratings with page views gives an idea of users willingness to rate articles. A user will rate 2.2-2.9% of pages they view, generating 0.84-1.1% usable ratings per page view.

Distribution of raw and filtered ratings were uneven. Raw data showing 78% of ratings occurring in the top 40% of the one to five scale, while filtered results showed 87% of ratings in the same 40%.

Figure 1 "A tally of the number system ratings, unbiased ratings, and manual ratings for each value"

A regular occurrence of positive ratings is found also to be true in eBay ratings, where members are very positive towards trading partners. Resnick and Zeckhauser (2002) suggests this is in keeping with social norms to treat others well. Dellarocas (2001) however puts it down to a "culture of praise", where members feel that the correct thing to do is to be kind and forgiving to other members. In doing so, users may also avoid rating bad pages.

Rated pages were manually assessed by the researcher to gain a baseline quality measurement (summarised in figure 1). Pages were judged on the potential usefulness of the information on a page (to an internal or external viewer), and the information content of the page. The scale used was a 1-5 rating. A better rating could be sought, either by building a more strict marking scheme, or consulting an expert in the field of instructional gaming, however the ratings generated are believed to be a suitable starting point for analysis.

The manual ratings (figure 1) show a more even distribution than the system generated ratings (also figure 1), with the mode slightly lower than the system generated ratings, and a minimum score of 2. These scores were compared with the system generated ratings to see how effectively the rating system performed in the wiki. No significant correlation was found between system ratings and manual ratings.

This study also analysed attention data. Attention data gives a measure of the amount of attention (measured in time) spent on a task (rating) or object (page). Attention data was collected from the Apache logs and processed by a script that identified individually a users actions, and by processing them chronologically, identified the amount of time between actions.

Analysis was made of time spent viewing pages, time spent rating pages, and a count of page views. This data was compared against system ratings and manual ratings. No significant correlation was found in this trial.

Survey

Two major findings in the logs were the low number of ratings, and the fact that a large number of ratings were identified as having a potential for bias. The questionnaire sought to understand the students' level of participation in the wiki, and what factors influenced it. It sought to understand how students viewed the rating system, how they used it, and possible biases. The questionnaire was conducted online and contained 57 questions, consisting of 51 numeric ratio entry questions, 1 short answer and 5 extended response boxes.

Students asked to participate in the wiki counted 13, and in total 11 students completed the survey. The 85% response rate is due to strong encouragement from the lecturer of the subject, and substituting the survey for what was originally an assessable task. Since the results of the survey were designed to provide guidance in interpreting the log results, the low sample size of 11, given the population size of 13, was not seen as a problem.

The ages of participants ranged from 20 to 32, with 20 the mode, an average of 24.4, and a median of 22. Participants were eight male and three female, with a roughly equal split of internal and distance education students, 55% internal, 45% external. Every respondent was completing an information technology or information science based degree. These results were expected for an information technology focused subject at CSU.

The survey sought to understand participants' confidence and ability with computing tasks. Students were asked to report their confidence in performing a number of tasks. Allowable responses followed a standard five-point Likert scale.

Computer self-efficacy results showed, as expected, a highly confident group of participants. Wiki self-efficacy questions showed that students learned all basic wiki skills required of them. A small number were not confident in more advanced topics, such as uploading files, and discussion signing, however all students possessed the minimum required skills.

Participants were asked about their previous experience and exposure to wikis. No respondent agreed that they knew what a wiki was before participating in this research, however three of the participants reported they were unsure. When asked if they had contributed to a wiki before, three were uncertain, while the remainder were confident that they had not. Realising that many people do not understand that there is a collaborative model behind Wikipedia, participants were also asked if they had ever used Wikipedia. Most (9) participants were unsure, while the remainder reported they had not.

When participants were asked if they had rated any articles, only six report they had done so. As reported above, some members may be hesitant to rate items in a small community and where benefits of rating are unclear. Rashid et al. (2006) found that the number of ratings contributed can be increased by illustrating certain benefits to the user. When asked if they understood how to use the rating system most responded positively, three were unsure, and only three were highly confident. Those who did not rate on average were slightly more confident in understanding the rating system than those who did rate.

The survey attempted to identify if there may have been any bias in ratings. When asked to honestly assess the accuracy of their ratings, participants were largely uncertain of the accuracy of their ratings. One respondent felt their ratings were not accurate, while only four showed any confidence in their own ratings. There was less confidence among participants of other users' ratings. eight of respondents displayed neither agreement or disagreement with the statement that other peoples ratings were accurate.

The exercises completed in the subject were intended to allow students to get to know each other, to work together, and generate friendship between peers. It was therefore pertinent to ask if participants felt they were likely to rate their friends' articles more favourably. A range of responses were received, from strong agreement (2), to strong disagreement (1). Overall two disagreed with the statement, while five expressed agreement.

Overall users of the wiki found ratings to be only slightly useful, one respondent found the system not useful, while five found it only slightly useful.

The survey asked students what factors they felt were important when rating articles. The three factors presented were visual quality, factual quality, and writing style. Students reported factual quality to be the most important factor, with all students expressing either agreement or strong agreement with the statement. The next most important factor identified was the visual quality. For this statement, a wider spread of responses was received from disagree to strongly agree. Writing style returned a similar spread of responses, with a slightly lower average, but still with general agreement. Very few students responded as unsure to any of these statements (one each for visual quality and writing style).

Respondents were given the opportunity to respond freely if they felt there were other factors they felt important. Two respondents made comments that language used in articles should be easy to understand, with one emphasising the importance of clear descriptions. Two respondents expressed that organisation of information should well thought out. Two respondents also said that articles should be written to allow for a wide audience.

One student provided a view on how ratings impact viewers.

Participants were questioned about the Pools of Online Dialogue (POD) exercises. Although there was no strong agreement with the statement, eight reported the pod exercises easy, while the remainder were undecided. Most respondents were confident in understanding the game they chose. Eight responded positively to the statement, with one of those a strong agreement. The remainder were undecided with the exception of one who found understanding the game a challenge. When asked what was found difficult about completing the POD exercises, the most common response was unfamiliarity with computer games, with three such responses. Two respondents reported finding a game to study a challenge, however no participant explained the reason for this. One participant also cited poor personal organisation and time management as an issue, while another found it a challenge getting to know the other POD members they were assigned to work with.

While students at CSU are being required to adopt various methods of communications technology, a wiki is a much more unrestricted and public space. Students were asked to rate how comfortable they were contributing to the wiki. All students responded positively, with the exception of one who declined to respond. Of those who responded, three students indicated they were strongly confident.

Students were asked to describe further their feelings working in such an open environment. Students were provided with a short list of words, excited, enthusiastic, proud, confident, indifferent, nervous, unsure, confused, and shy, and asked to provide three words, either from the list, or of their own, that describe how they felt contributing content to the wiki. In total eight responses were received, including about 19 single word answers, and three longer descriptions. These longer descriptions were simplified to single words, or short phrases, creating 23 words or phrases in total. These words and phrases were studied and grouped into five categories, representing different sentiments. These groupings were "strongly confident", "positive response", "neutral", "relief", and "doubting" (see table 2).

While some students used three words to describe similar feelings, there were also instances of varied feelings in combination. Three students expressed a mix, of a sense of achievement, and of relief, while another expressed enjoyment, while still being unsure about their contributions.

Students were asked to elaborate on their feelings by explaining their choice of words. A mixed response was received. Two responses expressed a lack of special enthusiasm of the wiki as a medium, that the only reason for participation was the completion of their required class-work. "It just felt like placing an article online for people to read", and is "[not] something to go crazy about". Another responded "it was an assignment. I probably wouldn't do it out of my own interest in Wikis[sic]".

Two respondents explained their relief came from having completed another university assignment task. One respondent however, as well as expressing relief, explained they were proud to have contributed to their POD group, despite they were unsure how their work would be received by others.

One reportedly excited and enthusiastic student was particularly interested in the outcome of the wiki, stating the activities were "something different", and that they were "keen to see how the Wiki would turn out". Two other students were enthusiastic and interested in the medium. One pointed out the importance of online communities, and that "to share our ideas and arguments is obviously a good thing". Another was pleased to have a space where they could "contribute [their] ideas and knowledge, sharing with other people", and likewise, to "learn different ideas and information from [other people]". "I felt enthusiastic, proud and confident when I posted my contributions".

Throughout the written feedback, there were several general comments about using the wiki. One student explains how they overcame problems when trying to understand and write wiki-text:

This comment shows a level of problem-solving ability, similar to that of hacker cultures (Raymond 2006), a personality believed to be common among wiki contributors.

Several responses, all positive in nature, were received regarding the personal learning experience of using the wiki. One expressed pride over using a wiki in this subject for the first time, while another rejoiced upon the "new learning methods", and knowledge they gained through using online tools.

One student summed these sentiments up in their response:

This response exemplifies several important features of wiki communities. Stallman (1999) explained how small contributions to wikis are important, and how wikis work as a communication and brainstorming technology, where users can contribute ideas a learn from others.

Concluding comments

Are there any mechanisms by which the community (or a subset) can provide some level of authority?

Two methods were trialled in this research in a small wiki. While neither provided accurate results in this setting, further research needs to be done with larger and established wikis. Further research may also develop alternate methods to test.

Can ratings made by users be reliably used to identify quality articles?

In small education wikis, no. Comparison of system generated ratings with manually assessed ratings showed little correlation. This analysis did find that the vast majority of system ratings were positive, with at least 78% of ratings found in the top 40% of the rating scale. Dellarocas (2001) and Resnick and Zeckhauser (2002) also found an overwhelming number of positive ratings when studying eBay ratings. In this trial, biases towards other members was identified as a possible factor reducing the accuracy of user ratings. The participant survey determined that this may significant factor. This fact may reduce in significance as the size of the community increases, and the likelihood of interaction between friends lowers.

Can attention data be used in an wiki environment to estimate the quality of an article?

In small wikis, no. Analysis of data showed some correlation between time spent rating pages, and their quality. This effect may however be useful in a larger wiki with a larger set of attention data. Apache logs as an attention metric also poses its limitations. Client side monitoring would provider richer data, allowing the researcher to have a better idea if the user was actually reading a page, had simply left the page open in the browser while working on another task, or closed the browser entirely.

How effective are wikis in educational settings?

Observation of the wiki found little interaction between participants. While it was rare to see any interaction where students posted their personal assignment tasks, most shared (group) pages experienced a high level of collaboration, and evolved quickly. These pages saw students adopt a shared presentation style, and in some cases, a form of group identity, where the group adopted a shared name.

The Georgia Institute of Technology, through the use of their CoWeb wiki found that in the right conditions, students form a group identity and shared respect. Students also find the wiki an enjoyable and efficient medium for discussion, learning, and sharing of ideas. Likewise this research discovered that some students identified the potential for shared learning and collaboration. Others saw it simply as a form of publishing.

The need for further research

Observations made during this research highlight the behavioural differences between small and large wikis. Wikipedia is a common target of study, due to the need to solve problems with its reliability. There is comparably little study done on small and medium wikis. There is a need for future work to study the dynamics and interactions of smaller wikis, determine a software tool-set that best fits their needs, so that they can be most effectively used. Further work is also needed to understand how best to design and manage small wikis in educational settings.

This research observed a high number of potentially "biased" ratings. These ratings were made by participants who rated the same pages they edited. Further work can determine firstly if user ratings produce reliable recommendations in larger wikis. Secondly if the high number of such ratings is a feature only of small wikis, and if such ratings are indeed inaccurate?

Results of this showed a high number of positive ratings. This has also been observed in communities such as eBay (Dellarocas 2001, Resnick & Zeckhauser 2002), however Malda (1999) through his work has created a system (and community) that generates more accurate ratings. Are there any features of the software system, its mechanisms, or the community that can account for these differences?

Analysis in this research showed inconsistencies between the manual ratings, and the system generated ratings. Is there a need for some form of training so that users understand how articles should be critiqued and assigned an accurate rating?

Established wikis form a culture, a with it a set of norms, expectations and guidelines (written or otherwise), that new members learn and follow when joining the community. Is training required to members in private (corporate) and educational wikis to act as a surrogate to cultivate an effective wiki in its early stages?

Bibliography