EVALUATING CRITICAL SUCCESS FACTORS OF DISTRIBUTED LEARNINGRussell BakerUniversity of Tampa [email protected]
eduAbstractDistributed learning presents universities and colleges with the ability to expand their reach into new markets and stay competitive and relevant in this dynamic information-based global economy. Through the effective use of distributed learning tools, location and cost are no longer barriers to earning a degree and enable universities and colleges to reach working adults and international students as well as further penetrate the traditional student market. This paper highlights the evolving transformation of Distance learning models to the evolving technology based distributed learning modes. While each institution has its own mission and goal for distance learning and distributed learning, there are certain things that need to be considered while developing or implementing a curriculum that involves education at a distance.
This paper explores distance learning from a macro perspective and suggests some critical success factors that will aid faculty and institutions in distance learning and distributed learning development. IntroductionDistance learning and distributed learning and their current platforms have a potential role to play in academic content delivery for educators globally. It can certainly do for higher education what the Gutenberg press did for the Bible. History tells us that until 1450 A. D., books were painstakingly copied by hand, a lengthy process that limited them to an elite few.
The combination of movable type, ink and press, however, greatly increased the distribution of the written word. Likewise, capacity in courses at Harvard, Stanford and MIT and Wharton is limited. With Distance learning, however, a Wharton Professor can teach students not just in Philadelphia, but now globally.
The Internet and the World Wide Web have revolutionized the way we teach, making it possible to move much, if not all, of what we used to do on paper into the realm of electronic media (Adams, 1998; Baker, 2003; Bender, 1995; Chimi and Gordon, 1997; Privateer, 1999). Defined in its most basic form, distance learning occurs when the student and the instructor are logistically separated. Considered from this perspective, distance learning is not a new concept to academia. Educational institutions have been providing distance learning courses in various formats for many years. Correspondence courses were offered as early as the mid-1800s (Sonner, 1999). As technologies developed, various types of distance learning have evolved along with the technologies. Universities have been providing directed and independent study distance learning courses utilizing videotapes and interactive television since these technologies became available. The advent of Internet technologies and their application to distance learning resulted in an explosive growth of distance-learning courses at the collegiate level.
According to U. S. Department of Education reports, distance-learning enrollments at the university level increased 70 percent during the period between 1995 and 1998 (Boehie, 2000). This growth has continued as more online courses became available through a variety of educational venues.
Proceedings of the 7th Annual Conference of the Southern Association for Information Systems256″Distributed learning encompasses the myriad of distance-learning venues throughout their spectrum of application. While true distance-learning courses are available from many institutions, many more provide distributed-learning courses. These courses contain many aspects of traditional, in-class courses, along with distance-learning components like online testing and email communication integrated into the course curriculum. The spectrum of application of distributed learning includes everything from a traditional course where the syllabus is posted online and email is used for communication to a nearly pure distance-learning course where an initial face-to-face meeting or in-class testing reflect traditional course design. It is our view that the winners in this dynamic knowledge-based economy will be those who can rapidly receive, filter, process and utilize information whenever and wherever it is desired or needed. While the electronic classroom still revolves around the primary classroom document, the syllabus, this document is no longer a static paper contract, but a living, dynamic electronic web page with multiple parts and pieces all linked together using hyperlinks (Papp, 2000).
The Internet has also made it possible to move the contents of the course on-line and new tools such as threaded discussion groups, chat rooms, and virtual lectures have made it possible to conduct a class entirely on line (Burns, 1999; Novitski, 1999). Thus, facilitating more ??? real world ??? inquiries that require students to use higher order skills. Our experience in developing distance learning course dates as far back as 1996. As early adopters of the Internet and World Wide Web (WWW) in our traditional on-campus classes, it seemed to be a logical and natural extension of what we were already doing. Much of our course material was already on-line and accessible through our course home pages or through our specific delivery systems. We both had detailed lecture notes and external links to other resources on-line and students were quite pleased with the amount of supplemental information provided to them. Our shared experience was our concern with how to duplicate the “ classroom environment” to an on-line setting and how students would respond to the lack of face-to-face, personal contact typical of an on-campus course.
Studies were conducted on two sections of the same course, one on-line and the other on-campus, investigating student perceptions and the level of learning taking place (Papp, 2000; Papp, 1999). Our findings suggest that there are several critical success factors that enable distance learning to thrive. These factors will be analyzed and discussed in detail in this paper.
The authors will also share their insights and experiences as they relate to these factors. It is our view that the proliferation of personal computers and the increasing penetration of the Internet are key contributors to the rising demand and success of distance learning. Internet use and diffusion is continuing to explode, growing at an unprecedented pace, reaching a 25 % market share in only 7 years, compared to 35 years for the telephone and 30 years for the microwave.
The Internet has grown exponentially since its birth in the mid-1990??™s. The diffusion of the Internet and computers in our daily lives is evidenced by the growing integration of the World Wide Web (WWW) and e-mail in university and college courses. It is suggested that: Over half of all college classes are using Internet resources as part of the syllabus, compared with 25% in 1997 and 15 % in 1996. Over one-third of college courses are using World Wide Web (WWW) pages for class materials and resources, compared with 8% in 1996 and only 4% in 1994 The percentage of classes using e-mail increased six-fold since 1994. Most class textbooks are complimented with student and instructor resources accessible from the Internet. Distributed learning today is more than just a phenomenon; rather it is quite a success for all stakeholders- students, instructors and institutions and private distance learning developers.
There has been a phenomenal growth in the number of instructors, universities and colleges that are seeking to develop education or course delivery systems that combine the best of traditional classroom instruction with the power of technology. This is illustrated by the growth in students enrolled in distributed learning and the number of distributed learning courses offered by two and four year institutions. In 1998, 710, 000 students were enrolled in distributed learning courses, this figure is expected to increase to 2. 2 million in 2003, representing a compound annual growth rate (CAGR) of 33 % (Papp & Kah, 2001). It is estimated that over 84% of four year colleges are expected to offer distributed learning courses in 2002, up from 62% in 1998.
Two-year colleges are also quickly moving into the distributed learning domain with over 85% expected to offer distributed learning courses in 2002, up from 58% in 1998 (Privateer, 1999). Proceedings of the 7th Annual Conference of the Southern Association for Information Systems 257″Thus, the purpose of this paper is derived from these issues. PurposeThe purpose of this paper is to identify and evaluate the critical-success factors for distance-learning. These critical success factors will then be integrated with the distance-learning evaluation matrix developed by Baker and Papp (2003) to provide a quantifiable measure of success. Distance-learning course(s) with curriculum designs that comply with the identified critical-success factors will be evaluated for success in attainment of course objectives using the Baker-Papp course evaluation matrix. Limitations of StudyThis study is subject to the following limitations: Courses tested are representative of a small private institution and may not be extensible elsewhere. Courses represent a distributed (hybrid) model rather than a truly asynchronous distance-learning course The courses were administered using Blackboard?„? and may not be extensible to other delivery systems Compliance with critical success factors was at the authors??™ discretionCritical Success FactorsThe acceptance of distance learning by some of the world??™s leading institutions–Stanford, Harvard, University of Pennsylvania??™s Wharton School of Business, Duke University??™s Fuqua??™s school of Business and MIT among others, is an indicator of the validity of distance learning as a an accepted pedagogy in higher education. In these days of enormous pressure on universities and colleges to attract new students and revenue streams, the benefits of distance learning cannot be ignored.
It provides them with the ability to expand their reach into new markets and stay competitive and relevant in today??™s digital and knowledge intensive economy. The benefits include increased access to education, increased access to best content, decreased cost and increased effectiveness (Land, 2002). Distributed learning courses and programs depend greatly on the effectiveness of their design, content, and mode of delivery. Currently, there exist a variety of approaches that universities and colleges are exploring to capitalize on distributed learning to shift into the demands of the evolving knowledge-based economy. The old perceptions of distance learning in the form of video and cable are being transforming by the rapid and dynamic advancement in technology. A primary concern of educators is insuring that courses delivered using distributed-learning or distance-learning pedagogies retain the effectiveness and quality of their in-class counterparts. Consequently, the following hypothesis is proposed: H1: Distributed learning course effectiveness is increased when the Baker and Papp Course Effectiveness Matrix (CEM) is used for course evaluation and design. A pre-requisite of a successful course in a distributed learning mode is content that is set high expectations from students through effective delivery of challenging subject matter in a manner that is motivating to students.
It is also important to carry content on user-friendly platforms that utilize multimedia tools, which students tend to be attracted to. It is also important that instructors delivering course content to be effective in using today??™s technology. This might require them to modify or shift their pedagogical paradigm and behavior. Consequently, the following hypothesis is proposed: H2: Student performance is elevated if higher levels of Bloom??™s taxonomy are incorporated in the course design as measure by the model. Several different alternatives for learning platforms are available to universities, including eCollege, WebCT, BlackBoard, or developing a customized web platform from scratch. Proceedings of the 7th Annual Conference of the Southern Association for Information Systems258″The easiest options for those who are not well versed in technology are eCollege (www. ecollege.
com) and BlackBoard (www. blackboard. com ) . Although the cost per student is higher, they do provide a high level of support and guidance. They will, given sufficient lead-time, transfer all your course materials from hardcopy format to web-ready format for you.
This is a very desirable option for those who do not want to learn the nuances of HTML and/or do not have a lot of lead-time to develop their course. BlackBoard (www. blackboard. com )l allows the course designer to quickly and easily put up a course web site with minimal development time and effort. Alternatively, the course designer can undertake the development of a complete website from scratch. This will provide the highest level of flexibility and customizability but also necessitates a strong background in technology and a willingness to spend considerable time in up-front website design and subsequent maintenance. Another consideration for faculty endeavoring to teach in a distance-learning environment is the suitability of the course for such an environment. Certain topics may lend themselves to this environment more readily than others.
Courses that depend heavily or completely on face-to-face interaction among students are much harder to conduct on-line. While technologies like chat rooms, threaded discussion groups and virtual meetings can bring students together over great distances of time and place, they still cannot fully duplicate the dialogue of the classroom. Faceto-face interaction is still a key component of many courses and while technologies like threaded discussion groups and chat rooms can replicate some of communication between students and faculty, there is no way to entirely duplicate the interactive classroom environment. With this in mind, course designers need to analyze their pedagogical approaches and determine if their courses can be successfully adapted for an on-line learning environment. Consequently, the following hypothesis is proposed: H3: Mode of course delivery does not affect attainment of course objectives when the course objectives are developed using the matrix. Effective use of distance learning technologies in the classroom can transform the learning process. Today??™s real world employer needs require students to use higher-order skills such as problem solving, collaboration, statistical analysis and simulation.
Applied projects require greater student initiative mandating that students take a more active role in their learning (Papp, 2003). Consequently, the following hypothesis is proposed: H4: The use of higher-level distance learning tools will result in increased levels of student satisfaction as measured by post-course evaluations. Current distributed learning tools give instructors powerful tools to monitor, guide and assess the progress of their students and to bring subject matter experts to interact virtually or deliver presentations to students (see Distributed/Distance Learning Tools in the Appendix). These learning information systems can be used to track student performance over time. Instructors can also use distributed learning tools to access resources to supplement instruction and exchange ideas with other instructors and professional experts in their domain (Baker, 2003). Consequently, the following hypothesis is proposed: Management Information Systems have become essential to the efficient functioning of successful businesses. In our view distance-learning tools in the form of learning information systems, the educational analogy to management information systems, have become an essential instructors aide in the classroom. Consequently, the following hypothesis is proposed: H5: Higher level Bloom Taxonomy objectives are more likely to be attained when higher-level distance learning tools are utilized Hypothesis Summary H1: Distributed learning course effectiveness is increased when the Course Quality / Effectiveness (CQE) matrix is used for course evaluation and design.
H2: Student performance is elevated if higher levels of Bloom??™s taxonomy are incorporated in the course design as measure by the model. H3: Mode of course delivery does not affect attainment of course objectives developed using the matrix. Proceedings of the 7th Annual Conference of the Southern Association for Information Systems 259″H4: The use of higher-level distance learning tools will result in increased levels of student satisfaction as measured by post-course evaluations. H5: Higher level Bloom Taxonomy objectives are more likely to be attained when higher-level distance learning tools are utilizedProposed MethodologyUsing the course evaluation matrix where appropriate, the hypotheses will be tested and validated using courses from a variety of institutions and delivery methods. Research methodology employed to test the hypotheses will vary according to the specific requirements of each hypothesis. H1 requires evaluation of course effectiveness based on a comparison of sections of the same course prior and subsequent to applying the CEM to course design and evaluation.
H2 requires evaluation of course effectiveness based on a comparison of sections of the same course prior and subsequent to redesigning the course incorporating higher levels of Bloom??™s Taxonomy into the course pedagogy. H3 requires evaluation of three course pedagogies, in-class courses, distributed-learning courses, and distance-learning courses. H4 requires measurement of student-satisfaction levels utilizing a post-course evaluation instrument subsequent to the H2 courses.
H5 requires multiple courses incorporating higher levels of Bloom??™s Taxonomy utilizing various categories of distributed/distance learning tools as defined in the Appendix. Course Evaluation ResultsData collection is in progress and preliminary results will be incorporated into the paper later this year. ConclusionsConclusions will be developed based on the results. It is expected that the hypotheses will be validated upon analyzing the data. Implications for EducatorsMany universities are beginning to look at DDL as an alternative means of content delivery and to reach nontraditional populations (Bialaszewski, et.
al., 1998; Fischer and OLeary, 1998; Russell, 1999). The creation of a DDL course has many rewards. Students like using a technology that they will employ in the working world, one that facilitates their learning and allows them to learn on their own time in their own way. They also like that they can attend the class when it is convenient for them and complete the assignments on their own schedule. This is particularly important for schools that face a great deal of competition in their area or enroll students from a wide geographic area.
Through distance learning, institutions can offer more sections and courses to students at times that are convenient for them. Since students are our customers, anything that can be done to retain and please them is seen as a positive step. From an instructor standpoint, several critical success factors can make the development and implementation of a distance-learning/distributed-learning course a fulfilling and rewarding experience.
While technology will always have its little surprises and unexpected problems, good preparation can go a long way toward making the transition to a distributed-learning environment easier. As the Internet moves further and further into the mainstream, distance learning will become a greater part of the educational process. While it will probably never replace the traditional on-campus class, it does provide alternative pedagogical approaches to learning and can make the student??™s educational experience more effective and enjoyable. RecommendationsThis paper represents research in progress and is designed to investigate not whether distributed/distance learning is appropriate, but rather an attempt to determine which factor(s) influence student-learning outcomes. It proposes a multi-class evaluation of several hundred students completing DDL courses and traditional on-campus courses in a variety of platforms. Continued research is necessary to validate the critical success factors proposed here to determine how and when distributed-learning environments should be employed and what facilitates their use.
Proceedings of the 7th Annual Conference of the Southern Association for Information Systems260″ReferencesAdams, C. ??? The Skills Audit Approach to Facilitate Undergraduate Learning??? Proceedings of the 13th Annual Conference of the International Academy for Information Management, Helsinki, Finland, December 1998, 211-16. Baker, R. ??? A Framework for the Evaluation of Distance Learning Courses??? Journal of Distance-Learning Administration, 6, 2, 2003. Baker, R. & Papp, R. ??? A Multivariate Matrix for the Evaluation of Distance-Learning Courses???, Proceedings of the European Conference on Information Systems, Naples, Italy, June 2003.
Bender, R. Creating communities on the Internet: Electronic discussion lists in the classroom. Computers in Libraries, 15: 5, 1995, 38-43. Bialaszewski, D., Burns, J.
, Dick, G., Papp, R., & Pencek, T. “ Web-Based Teaching: Past, Present, and Future.” Proceedings of the 13th Annual Conference of the International Academy for Information Management, Helsinki, Finland, December 1998, 192.
Burns, O. M. ??? The virtual lecture: An essential component of web-based course delivery??? Proceedings of the 14th Annual Conference of the International Academy for Information Management, Charlotte, North Carolina, December 1999, 387. Chimi, C.
& Gordon, G. ??? Using innovative information systems techniques to teach information systems??? Proceedings of the 12th Annual Conference of the International Academy for Information Management, Atlanta, Georgia, December 1997, 161-66. Fischer, D., & OLeary, A. Web-Based Distance Learning. Proceedings of the 13th Annual Conference of the International Academy for Information Management, Helsinki, Finland, December 1998, 85-89.
Land, D. ??? Experiencing the Online Environment??? USDLA Journal, 16??? 1, 2002, 1-6. Novitski, J. Asynchronous learning tools–what is really needed, and used Proceedings of the 14th Annual Conference of the International Academy for Information Management, Charlotte, North Carolina, December 1999, 189.
Papp, R. ??? Distance Learning: Its the Message, not the Medium??? Proceedings of the 2000 Northeast Decision Sciences Institute, Atlantic City, New Jersey, March 2000. Papp, R. & Kah, M. ??? Distributed Learning Environments: Pedagogical Concerns??? Proceedings of the 2001 Southern Association for Information Systems, Savannah, GA, March 2001, 248-56.
Privateer, P. ??? Academic technology and the future of higher education??? Journal of Higher Education, 1999. Russell, T. ??? The no significant difference phenomenon???, Office of Instructional Telecommunications, North Carolina State University, 1999. Williams, J. ??? Critical Success Factors of Computer Information Systems Distance Learners??? Proceedings of the 6th annual conference of the Southern Association for Information Systems, Savannah, Georgia, March 2003, 431-36. Appendix ??“ Distributed/Distance Learning Delivery ToolsDistance learning courses are by definition not face-to-face. Additionally, they may be either synchronous or asynchronous.
The differences between traditional in-class courses and distributed/distance learning (DDL) courses create several factors which need to be addressed in DDL course design. First, traditional courses are face-to-face Proceedings of the 7th Annual Conference of the Southern Association for Information Systems 261″synchronous courses. This means that learning occurs with the instructor in front of the learner and instruction and learning occur simultaneously and in the same place. This personal interaction between the two gives the instructor the opportunity to provide feedback, direction, and to observe learning activities. It gives the learner the opportunity to solicit feedback and receive responses and directions in real time. Second, traditional courses place the learner with other learners.
Learners are able to draw from each others experiences and interact in groups. Third, the requirement to attend class creates a responsibility for the learner and provides a source of accountability, possibly increasing his/her motivation to perform the required tasks. To develop procedures for addressing these factors, an understanding of the tools available for distance-learning instructional methods and course design is needed.
A variety of web-based tools and course design strategies including the following, are available (Ferguson, 2001; Baker, 2003): 1. Syllabus / course outline posting While syllabus posting is available for both DDL and non-DDL courses, the absence of an instructor to review the syllabus with the learner requires an easily understandable and comprehensive syllabus. 2.
Video classroom Streaming video has replaced video tapes as the tool of choice for lecture delivery in DDL courses. Video must be interesting, engaging, and worthwhile. Videos should not simply restate what is available in written materials.
A primary consideration for streaming video is the bandwidth available to the learner. While high speed connections are usually available on campus, Students who rely on dial-up connections will be at a significant disadvantage when using streaming video. Provisions for downloadable or CD-Rom video files should be made to prevent this problem. 3. Course Notes Course notes can be posted to the web to supplement video lectures and required readings.
4. Course Reference Materials, Readings, Cases Supplemental reading materials can be posted to the web for students to download. Links to other websites are frequently provided in references. 5. Chat Rooms Structured chat rooms conducted by the instructor provide group discussion on course activities and assignments. Using real-time chat, the instructor can ask questions during in a similar manner to the traditional classroom. Audio chat, a web-based tool that functions similar to teleconferencing, using a tool like Microsoft??™s NetMeeting ?® has an advantage in that it is more spontaneous than text-based chat and not dependent on the learners keyboard speed.
Chat also allows the instructor to provide immediate feedback to learner questions, evaluate learner participation, and take attendance. 6. Email Email allows students to asynchronously communicate with their instructor. Learners can ask questions and send assignments to the instructor. The instructor can use email to send evaluated assignments back to the learner 7. Bulletin Boards, Group Discussion Boards, Digital Drop Boxes These tools allow the learners to collaborate on projects, exchange ideas and participate in group activities.
8. Online Testing Online testing procedures allow the instructor to design evaluation instruments comparable to any form of paperbased instrument. Multiple choice, true or false, matching, and fill-in-the-blank questions can be automatically graded and posted.
Long or short answer essay questions can also be used. However, essay questions must be graded by the instructor. 9.
Interactive Activities Interactive activities provide a method of having the students practice desired behaviors. Click and drag techniques can be used to assemble components online (for example atoms into a molecule). 10. Feedback Proceedings of the 7th Annual Conference of the Southern Association for Information Systems 262″Specific provisions must be provided to insure students receive sufficient feedback. A frequent criticism of DL is its disembodied nature restricts feedback leaving learners feeling abandoned. Instructors must be trained to promptly respond to emails. Virtual office hours can be held using chat. Computer graded exams should have provisions for giving the student correct answers to the questions answered incorrectly.
Provisions for both asynchronous and synchronous feedback should be provided in course design. Asynchronous feedback occurs when the individual requesting the feedback (the student) experiences a time delay before feedback is received. Synchronous feedback occurs when the feedback response immediately follows the question or request with no time delay. 11. Virtual Classroom An online, interactive class session between students and instructor.
Simulates much of the interaction found in traditional face-to-face classrooms. Frequently incorporates other web-based tools including audio chat, video classroom, whiteboarding, etc. 12. Whiteboarding The ability to write and draw on an electronic board during a virtual classroom session. The DDL delivery tools can be segregated into three distinct categories.
Passive tools are those that deliver information to the learner with little or now action on the part of the learner. Sometimes called digital reading rather than digital learning, These tools do nothing more than post readings that the student can access. Listed DDL tools in this category include syllabus or course notes posting, video (video tape or streaming video), and course material posting. Since these tools require little if any action on the part of the learner they are considered low-level tools. Active tools require action on the part of the learner to achieve course objectives.
Activities like online testing or posting assignments that require further web work fall into this category. Bulletin boards, digital drop boxes and threaded discussion boards are additional examples of active tools. Interactive tools are the highest level of DDL tools. Interactive tools require the learner to interact with the learning site in real time. Interactive tools allow the course designer to create activities or experiences to which the user responds and receives immediate feedback.
Click and drag, online questions with answers and the opportunity to retry incorrect answers, and virtual classrooms are examples of interactive tools. The primary differences between the three levels of DDL tools is the degree of learner involvement in web-based activities and the immediacy of feedback provided to the learner in response to learner participative activities. Proceedings of the 7th Annual Conference of the Southern Association for Information Systems263
