geistlogistic

The Dutch Higher Education Foundation -  SURF invited Wolfgang and me to a seminar on Learning Analytics, where we presented our Learning Analytics framework and a questionnaire that is build on top of it.
They brought some interesting parties from different educational institutions (schools -> universities) and some companies together.
One observations was that the companies mainly focus on business analytics for the educational sector and the management of an educational institute, whereas the educational designers and researchers tools presented to support students and teachers in improving learning and teaching.
That reminds me on the TEL recommender systems that were also applied in the beginning like in the MovieLens system and even used their datasets. They mainly recommended content from related persons without considering context of learners like learning goals or prior-knowledge levels to recommend peers, learning activities, or learning paths.

Wolfgang and I tried to paint the big picture of Learning Analytics with the framework and give some practical examples. Both parts of the audience (the companies and the educational institutes) found the framework rather useful to shape the goals of Learning Analytics applications.
What became clear from the educational institutes is that we need to provide solutions for the big players (Moodle, Blackboard or Sharepoint) when we want to run any experiments on Learning Analytics with them. Most of the educational providers in the Netherlands use one of these systems and any learning analytic tool needs to address them. Thus, after prototyping and having valuable outcomes you need to address one of the big systems to disseminate your learning analytic solutions to the stakeholder

Below you can find our presentation that received 650 clicks in 3 hours. That was really impressing. I received the following mail from slideshare:
“Turning Learning into Numbers – A Learning Analytics Framework” is being tweeted more than anything else on SlideShare right now. So we’ve put it on the homepage of SlideShare.net. 

Well done!

- SlideShare Team

Another indicator that Learning Analytics is a very hot topic.

Below you can find the presentation. Special thanks belong to Peter Kraker who provided us with his twitter visualization tool that enabled me to show some real time reflection examples of the seminar on Learning Analytics. Thanks Peter!

The first presentation this year was given at the Learning Network seminar series at CELSTEC. Special guest was Wolfgang Reinhardt from the University of Paderborn who provided his view on data science in relation to awareness improvement for knowledge workers. The dataTEL presentation is based on the ECTEL10 but it also includes the latest developments on TEL recommender after the dataTEL System Marketplace at the RecSysTEL workshop what was a point of change in the research community. In the presentation below I show some of the changes and the new developments. Surprisingly, we had a very controversial discussion, more controversial than at the ECTEL conference.

I sum up the comments shortly:

• The collected datasets are far to small to conduct proper information retrieval on it.

But maybe they present the majority of datasets that are available in education so we have to adjust our techniques.

• Not all of the datasets are related to learning, esp. Mendeley.

Yes, that is correct but why should we limit our self as it is already quite a challenge to get a datasets.

• The privacy protection right will stay so we will never have the opportunity to use the student data from a LMS like Blackboard for further analysis.

But we could ask the students if they agree to give us there data for research purposes and be very explicit what we want to do with it. European Schoolnet did the same with the users of the eTwinning project.

The RecSysTEL workshop that was sponsored by dataTEL and the Organic.Edunet project was really an exciting event.  It was a big step forward for the RecSys research in TEL esp. on: 1. Extending the research community on RecSysTEL, 2. Changing the way RecSysTEL research will be conducted in the future.

Regarding 1, we took advantage of the lucky situation that the ECTEL and the ACM RecSys conference were taking place in the same week in Barcelona. A great opportunity to combine both research communities in one workshop. In the end we created a kind of own mini conference with some core people that attended both workshop days and a wider audience from both research communities that attended one particular day. People traveled between the ECTEL location and the ACM RecSys location so we did not only link the people virtually . Furthermore, we had a keynote from Joseph Konstan, Grouplens research and Kris Jack form the Mendeley startup. Joseph keynote talk was highly appreciated by the ECTEL community and really had an impact on the ongoing research in TEL recommenders.
Kris presented the Mendeley reference system and their datasets that they released in cooperation with the our dataTEL dataset challenge. We recorded both talks and will broadcast them soon.

Regarding 2, one special focus of the RecSysTEL workshop was on datasets that can be used for RecSys research. In order to collect relevant TEL related datasets, the first dataTEL challenge was launched as part of the RecSysTEL workshop. In this sub-call of the workshop, research groups were invited to submit existing datasets from TEL applications that can be used for research purposes on recommender systems for TEL. We opened the first ‘dataTEL system marketplace’ at the 1st day of the workshop.
You can find an overview of the presented datasets in a posting of Guenter Beham in the dataTEL group space at TELeurope.eu.
The most pressing topics in this session were the need to find a standardized meta data structure to exchange datasets, how we can deal with privacy and legal protection rights, anonymization of datasets, pre-processing of datasets, and shared evaluation metrics to compare the effects of TEL recommender systems. This very interactive session was a big step for the community and kept the people in a very crowed room without any window until 6:30 pm, whereas the sun was shining outside in beautiful Barcelona!

It became clear that the ultimate goal of the RecSysTEL research is a research infrastructure where researchers can find well documented and version controlled datasets of different research institutes ranging from formal to informal learning applications. Every RecSysTEL research needs to reference a publicly available dataset to make its results repeatable and comparable and to describe its contribution to the improvement of learning.
The current research practice is mainly based on small-scale experiments in which a few learners are asked to rate the relevance of suggested resources in a controlled experiment. Whereas such experiments offer valuable insights into the usefulness and relevancy of recommender systems for learning, stronger conclusions about the validity and generalizability of recommender experiments are needed in order to create a theory of personalization in TEL.

A theory of personalization in TEL needs more verifiable and repeatable experiments that allow the comparisons of results based on datasets that capture learner interactions. A dataset collection / infrastructure could support researchers to create repeatable experiments to gain valid and comprehensive knowledge about how certain recommender system algorithms perform on certain datasets in a particular learning setting.

The impact of the workshop on the research community is already visible on an increasingly amount of comparison studies that are currently conducted by different research units. The current studies are still quite basic as they apply traditional collaborative filtering algorithms on different educational datasets and report the results of these studies. BUT this is the right way we have to follow to gain valid knowledge on the impact of recommender and personalization of learning.

Based on the big success of the workshop we organized a follow-up workshop at the upcoming ARV2011 in March. Again we will have a two-day workshop, again we will have exciting keynote speakers, and again we will have very exiting contributions, so good signs for another outstanding event.

Pressing topics are:

• publicly available data sets for educational systems
• dealing with legal protection rights towards data sets on a European level
• privacy preservation for educational data sets
• methods of effective anonymization of educational data sets
• management and pre-processing procedures for educational data sets
• future scenarios for educational data sets
• impact of educational data sets for learners and teachers
• mash-ups based on educational data sets
• recommender approaches that are based on educational data
• evaluation methodologies and metrics for educational recommender systems

Besides these topics we are planning a 1st dataTEL competition where different research units will have the opportunity to compete with their algorithms on specific educational datasets. Therefore, we are in contact with Shlomo Berkovsky (ICT Centre, Hobart) who organizes the CAMra competition on context aware recommender systems. With the dataset competition we want to attract also people from other research communities like ACM RecSys but also EDM (Educational Data Mining) and other information retrieval communities to work on educational datasets and increase the knowledge base on personalization technologies in TEL.

Latest stats

• 40 users are registered (+ 25%)
• 4261 items are available in the system (+ 26%)
• 220 items are rated (+ 71%)
• 713 recommendations are offered (+ 38%)

Those are great numbers ReMashers!

Latest News

Luckily David Wiley visited us at OUNL, Marco and I had the opportunity to talk to him about the Open Content issues. We also presented to him the ideas behind ReMashed and further development plans. David was pretty excited about the system and compared it with their OCW finder. We were happy to find him a day after our meeting as a member of ReMashed.

Other observations

The rating based algorithm seems to come to a threshold that makes its recommendation more reasonable. ReMashed user have now the opportunity to experience how the algorithm works in small communities. For example, I get now recommendations for items I never rated but my neighbors did. As five ReMashed users sitting in the famous ApeCage at OUNL we share our experiences with the system with each other. I complained about one rating-based recommendation I received and was wondering who could have rated such an item. My colleague Marco told me that he rated it just for fun and now I have to live with this recommendation. Arrgh! (Reminds me to add the rating of recommended items functionality to the system).

More room for improvement

I got various requests how people can rate an item that is recommended to them when they like it. As I said we are working on that update but it will come with different other updates. For now you have to follow the recommended item to its original source, tag it in delicious and rate it afterwards when it appears in ReMashed (Uhhh bad user interaction). In a future future release we have to create the possibility to add sources from ReMashed directly to delicious we added it to the ToDo list.

Export recommendations via RSS

Another request is the export of the recommendations via RSS. Three people already asked if they can subscribe via RSS to the recommendations of ReMashed. That’s a nice and important idea that we will keep in mind for future development and also added it to the ToDo list.

Future development

Talking about future development, I can present an initial mock-up of the next release were users can specify additional Web2.0 sources (twitter & Youtube) and define interests fields (learning goals) with a self-assessment slider. The interests fields will be taken into account for future recommendation regarding personal competence development. The idea is that a user can explicitly specify 1 to 3 interest fields (learning goals) and his related competence level between 1-5.  An additional recommendation algorithm will be triggered by this goals and present relevant items for future competence development to the user in a separated box.

As a smart edition the interest fields will be powered by an auto completion algorithm. This algorithm will be fed with learning goals of other users and tags that are available in the system. Using auto completion helps to support users to use a shared vocabulary regarding their competence development. Later on users can be grouped according to their shared learning goals. More tailored recommendations can be created through recommending resources of users that are on a higher competence level than the current user.

Using domain data sets to cover cold-start and add additional techniques

Through the current pilot we came across a new idea to create different domain data sets with rated items to cover the cold-start problem of the recommender. The running pilot is a good example of a Technology Enhanced Learning data set. If we could run different pilots in various domains we could create several domain data sets (medical, engineering, education, security etc.)  and apply them to cover the cold-start and add additional technologies like Latent Semantic Analysis.

If anyone is interesting in cooperating with us in additional ReMashed pilots don’t hesitate to contact us.