Creating the eDesign Assessment Tool (eDAT) to Represent and Evaluate Online Distance Learning Designs: A Mixed Methods Study

There are concerns that retention is lower in ODL (online distance learning) than in face-to-face courses and that this has a negative impact on both student and institution. Learning design has been shown to have an impact on retention, but a tool for a variety of developers, designers and tutors to consistently describe and compare learning designs of ODL courses remains elusive. This study argues that the lack of a common terminology and different tutor perspectives about learning hamper both development and representation of ODL designs.
Existing research suggests that a complex mixture of student demographics, characteristics and skills, as well as course and institution features including level, subject and amount and type of tutor support may impact on ODL student retention. In particular, research suggests that activities that include interaction and feedback in ODL have a positive impact on retention. However, much of this research has been conducted using a range of student surveys that do not allow for comparison to retention data, or across courses or institutions due to their subjective nature. There has been little research on the impact of tutor perspectives on developing, representing and sharing learning designs.
This study examines the creation of the e-Design Assessment Tool (eDAT) that represents and quantifies interaction and feedback activities so they can be compared to retention and other learning data for a course. A mixed methods approach was used to:
a) test the effectiveness of existing terminology for categorising learning activities using a content analysis methodology by trialling sets of terms with a sample set of ODL course activities.
b) identify tutor perspectives about learning and teaching using repertory grids based on personal construct psychology, and exploring the impact of these perspectives on the different meanings and uses of learning activity terminology.
The content analysis testing of learning activity terminology was challenging. The pilot studies had low inter-rater reliability, suggesting difficulties in independent rating of existing learning design terminology. However, the final eDAT tool created through data collected for this study used terms that did lead to a greater level of inter-rater reliability.
A significant contribution of this study is the use of repertory grids to gain insight into the issues relating to the development of a quantitative tool. The repertory grid interviews indicated that there were significant differences in the ways tutors and raters understood and used key educational concepts including interaction and feedback, and that there was a variety
of vocabulary used when describing learning activities. This study argues that tutor perspectives impact on designing, representing and evaluating ODL courses in a way that has implications for learning design and for professional development of tutors.
The final eDAT builds on and develops existing learning design representation and evaluation tools, but utilises more consistent terminology. Thus it offers a simplified approach to pedagogic guidance in the form of quantification of interaction and feedback activities in a course, and embeds reflection on tutor perspectives underpinning the design to support sharing and reuse. This combination will lead to better ways to represent learning designs, as well as providing a method for gathering learning analytics data useful for the comparison of learning designs to student retention data and thereby improve practice in ODL.