geistlogistic

Below the presentation of the paper written by Wolfgang Greller and myself and our international survey on Confidence in Learning Analytics at the LAK12 conference, Vancouver, Canada

Hi folks, here you can find the introduction slides for the LALD workshop 29th of April 2012 at the LAK12, Vancouver, Canada. Looking forward to it, as we made very good experiences with the PMI rating and the Grand Challenge task in previous dataTEL workshops.

In the third delivery of our survey series on Learning Analytics we focus on the results of the survey around the subdomains “Educational Data” and “Applied methods and Technologies” of the Learning Analytics framework.

Educational Data:

The section on data investigated the parameters for sharing datasets in and across institutions. The potential of shareable educational datasets as benchmarking tools for technology enhanced learning is explicitly addressed by the Special Interest Group (SIG) dataTEL of the European Association of Technology Enhanced Learning (EATEL). Sharing of learning analytics data is impeded by the lack of some standard features and attributes that allows the re-use and re-interpretation of data and their applied algorithms. For researchers, the most important feature was the availability of added context information (n=43, means 3.42) with a maximum value of 4 on the Likert scale. Perhaps, equally unsurprising was that for the manager group sharing within the institution (n=16, means 3.63) and anonymisation (n=19, means 3.53) were the most important values. Teachers, on the other hand, valued context (n=52, means 3.42) and meta-information (n=47 means 3.47) the most. At the other end of the spectrum, version control was the least important attribute across all constituencies (n=106, means 2.93). However, despite ‘version control of educational datasets’ was ranked the lowest, we still believe that this will play an important role in an educational data future. Version controlled datasets will offer additional insights into reflection and improvements through learning analytics by comparing older and newer datasets. Graph 6 illustrates the importance of the given data attributes. Note that the notion of “important” outweighs the “highly important” overall, which results in a lower means value.

To get an idea of existing educational data, we asked participants about their institutional IT systems. For learning analytics, the landscape of data systems will play an important part in information sharing and comparison between institutions.
In the tertiary education sector alone (Further and Higher Education), 93.9% (n=92) reported an institutional learning management system, which made this the most popular data platform by far. This was followed by a student information system 62.2% (n=61) and the use of third-party services such as Google Docs or Facebook 53.1% (n=52). Table 2 below shows a summary inventory of institutional systems in use across all sectors of education covered in our demographics.
We assume that the more widely available a type of system is, the more potential it would hold for inter-institutional sharing of data, which could be utilised for comparison of educational practices or success factors. However, such sharing would depend on the willingness of institutions to share educational datasets with each other. When asked this question, a majority of people (86.6%, n=71) were happy to share data when anonymised according to standard principles.

What is slightly contradictory is that people who indicated before that anonymisation was not an important attribute for data are less inclined to share (n=18, 83.3% yes : 16.7% no) than people who felt that it was highly important (n=40, 92.5% yes : 7.5% no).

Methods and Technologies
Learning analytics is based on algorithms (formulas), methods, and theories that translate data into meaningful information. Because these methods involve bias [1], the questionnaire investigated the trust people put into a quantitative analysis and in accurate and appropriate results. Within the 100% rating range, where 100% would indicate total confidence and 0% no confidence at all, the responses were located at mid-range. Among the given choices, slightly higher trust was placed on the prediction of relevant learning resources. This may be due in analogy to the amazon.com recommendation model, which is well-known and widely trusted. Other recommendations, such as predictions on peers or performance were rated rather low. The percentage on the horizontal axis in graph 7 below shows the level of confidence.

One comment criticised that it was “disappointing that you included institutional markers, rather than personal ones for the learners, e.g. while learning outside the institution, which in my view are much more important and interesting”. We are not aware that the questions actually reflected an institution-centric perspective. At the same time, we still remain sceptical that analytics might currently be able to seamlessly capture learning in a distributed open environment, but mash-up personal learning environments are on the rise and may soon provide suitable opportunities for personal learning analytics.

In our next blog posting we will focus on the subdomains “Constraints (Privacy and Ethics)” and new “Competence” that are needed for Learning Analytics.

Mendeley is recruiting a Marie Curie Senior Research Fellow. Your primary responsibility will be to ensure that Mendeley’s research catalogue (i.e. collection of articles) is of high quality. Mendeley has crowdsourced the world’s largest research catalogue with over 50 million unique articles contributed by almost two million users over a period of four years. With your expert knowledge in data technologies and algorithms, you will take ownership of this catalogue, and work on innovative techniques for improving its quality. Your work should result in a cleaner, better structured and more scalable catalogue.

This position is part of the TEAM project (http://team-project.tugraz.at). You will spend 1 year in Mendeley’s London office before spending 1 year at TU Graz, the Knowledge Management Institute (http://kmi.tugraz.at/), Austria, collaborating with a top-class team. You will be passionate about working with large scale data collections and take pride in producing high quality data.

Description

Responsibilities

Ensure that the research catalogue is of high quality
Understand, maintain and help develop current crowdsourcing system
Disseminate results from your work both internally and externally

What you’ll be doing

Crowdsourcing a homogeneous catalogue from heterogeneous data sources, using modern data techniques
Identifying data sources, judging their appropriateness and working with data engineers to import them into the catalogue
Working with Data Engineers and Platform Team to make reliable/scalable systems
Working with Data Architect to ensure coherent data mapping, ontologies and schemas
Working with Mendeley’s Chief Scientist in contributing to solving data problems outside of the scope of catalogue crowdsourcing
Working 1 year from Mendeley’s London office, followed by 1 year in TU Graz before returning to London, with regular travel between both locations

What you should bring

PhD in the field of Computer Science or 4-10 years of full-time research (following first publication)
Expert knowledge of text and document processing, with strong machine learning background
Experience working with large-scale catalogues
Database integration experience
2+ years of Java programming; can independently prototype solutions to problems
Experience with big data technologies (e.g. Hadoop, MapReduce, NoSQL)
Unix skills, preferably Linux
Fluent spoken and written English
Strong presentation skills in communicating with experts and novices

What we offer

Salary of £50k per annum + stock options
No out-of-hours support expected
25 days holidays
Company benefits such as: cycle to work scheme, childcare vouchers, BUPA (private healthcare), Friday beer o’clocks (snacks and drinks on the house), free breakfast, monthly team night’s out, annual events (Christmas party and summer barbecue)
Working in a great environment in a central London office with roof terrace

TEAM-specific restrictions

Nationality: The researcher may be a national of a Member State of the Union, of an Associated Country or of any other third country
Mobility: At the time of selection, the researcher must not have resided or carried out his/her main activity in the country of the beneficiary home organisation for more than 12 months in the 3 years immediately prior to his/her selection under the project. International European interest organisations or international organisations.
The appointed researcher must not have spent more than 12 months in the 3 years immediately prior to the selection by the home organisation in the same appointing organisation.

If you are interested, send your CV and cover letter to jobs [at] mendeley [dot] com. If you are selected for an interview, we will let you know within two weeks.

Since our last blog posting on the Learning Analytics survey our framework became now also cited (next to JISC) in the EduTech wiki.

Our blog posting on the results is organized along the lines of the six dimensions of the learning analytics framework (see the previous posting). We paid special attention to mapping opinions against institutional roles in order to identify any significant agreement or discord in each of the dimensions.

In this follow-up blog posting on our Learning Analytics (LA) survey we describe some results regarding the ‘Stakeholder’ and ‘Objectives’ subdomains of the LA framework. The full overview of question items and answer types can be found in our dspace repository.

Stakeholders:

In this section, we wanted to know: (a) who was expected to benefit the most from learning analytics, and, (b) how much will learning analytics influence specific bilateral relationships?

Regarding the prioritisation of the stakeholder of learning analytics, the majority of respondents agreed that learners and teachers were the main beneficiaries of learning analytics where 1 was the highest score on the Likert scale. The weighting of the 155 responses shows that learners were rated highest at 1.9 mean rank, followed by teachers with 2.1. However, the ranking distribution and standard deviation for learners was higher (1.12) than for teachers (0.88). Institutions came in third place with an average rank of 2.6. There was also substantial contribution to the ‘other’ category with suggestions for further beneficiaries. Among those and most prominent were government and funding bodies, but also employers and support staff were mentioned.

Graph 1 above illustrates the outcomes of question (b) and confirms the findings of question (a) above. The peaks identify the anticipated intensity of the relationship. Relationships with parents are not seen as majorly impacted, which is probably due to the fading influence parents have in tertiary education. It would be interesting to complete this picture with more responses from the K-12 domain. The highest impact is seen in the teacher – student relationship (83.5%, n=111, of respondents emphasised this), whereas the reverse student – teacher connection is strengthened slightly less (63.2%, n=84). Only less than half the participants see peer relationships as being strengthened through learning analytics: learner – learner by 45.9% (n=61), and teacher – teacher by 41.4% (n=55). At roughly the same level comes the relationship between institution and teachers (46.6%, n=62). The relationships of teachers that are expected to be most widely affected, followed by learners, institutions, and parents at a minimal level.

Objectives:

In this section, we asked participants in which way learning analytics will change educational practice in particular areas. Of the total answers given in all 13 areas (n=1543), collected from 119 participants, only 10.8% of responses anticipated no change at all. On the other hand, the remaining responses left it open whether the expected changes will be small (43.8%) or extensive (45.4%).

Looking at the individual areas (cf. graph 3 above), the highest impact was expected in more timely information about the learning progress (item 2), and better insight by institutions on what’s happening in a course (item 8). On the bottom end were expectations with respect to assessment and grading (items 6 and 5), where the least changes were anticipated.

Further, we contrasted the importance of three generic objectives for learning analytics: (a) reflection, (b) prediction, (c) unveil hidden information. 47% (n=61) of the respondents felt that stimulating reflection in stakeholders about their own performance was the most important goal to achieve, while 37% (n=48) expressed the hope that learning analytics would unveil hidden information about learners (cf. graph 4 below). Both are not necessarily in contradiction to each other, since insights into new information can be seen as motivator for reflection. However the case may be, only 16% (n=20) favoured the prediction of a learner’s performance or adaptive support as a key objective.

When looking at these objectives from the perspective of the different roles of participants, we find that teachers show a fairly equal interest in unveiling hidden information 44.6% (n=25), and in reflection 37.5% (n=21). This is a reasonable finding as many teachers expect learning analytics to support them in their daily teaching practice by offering additional indicators that go beyond reflection processes. On the other hand, 60.4% (n=29) of researchers indicated a clear preference for reflection.

Translated into technological development, the expectations favoured more adaptive systems (highest rank), followed by data visualisations of learning, and better content recommendations in third place. Further interesting suggestions were “learning paths/styles adopted by students”, the clustering of learning types, and applications for the acknowledgement of prior learning.

A further question surveyed the perception of learning analytics being a formal or less-formal instrument for institutions. In two intermixed sets of three options, one set represented formal institutional criteria: mainstream activities, standards, and quality assurance, all relating to typically tightly integrated domains that are governed by institutional business processes and strategies. The other set contained three less-formal and less monitored areas of pedagogic creativity, innovation, and educational experimentation.

All three items represented individual choice of staff members to be innovative, experimental, and creative in their lesson planning and teaching activities. As indicated in graph 5 below, among the 129 responses, there was a noticeable preference towards less formal institutional use of learning analytics at a ratio of 55:45 per cent. Quality assurance ranked highest in importance among the formal criteria, whereas innovation was seen as most important aspect of all criteria.

One participant summed up the situation of these findings in the following statement: “It would be easy for learning analytics to become a numbers game focused on QA, training/instruction and rankings charts, so promoting its creative and adaptive potential for lifelong HE/professional-life learning is going to be key for the sector – unless learning analytics people want to spend all their lives doing statistical analysis?”

In the next blog posting we focus on the results of the survey around the subdomains “Educational Data” and “Applied methods and Technologies” of the Learning Analytics framework.

We recently submitted the final version of a book on “Recommender Systems for Learning” (#RSFL) to Spinger (to appear soon in 2012) that focus on the past 10 years of research on recommender systems in technology-enhanced learning (TEL).

We introduced recommender systems and compared them to relevant work in TEL like adaptive educational hypermedia, learning networks, educational data mining and learning analytics. Then we emphasised on TEL as a recommendation problem, discussing how the recommendation problem is defined, which the recommendation goals are, and what the recommendation context usually covers as context.

We reviewed existing TEL datasets that may be used to support experimentation and testing, as well as discussed about how they can drive relevant research. We reported an extensive analysis of existing recommender systems that can be found in the literature for educational applications. And finally, we reflected on some major challenges that we see as important to be faced in the years to come, also outlining some potential directions of future research.

All the bibliography covered by this book is also available in an open Mendeley group with the same name “Recommender Systems for Learning“and will continue to be enriched with additional references. We would like to encourage the reader to sign up for this group and to connect to the community of people working on this topic, having access to the collected bibliography but also contributing pointers to new relevant publications within this very fast emerging research field.

Science 2.0 deals with the involvement of the web in science. It spans from the utilization of Web 2.0 tools and technologies in research to a more open and sharing approach to science. Some definitions of Science 2.0 even include notions of a methodological change due to the abundance of data, and the nature of the socio-technical systems on the web. For this special track, we would like to address four issues in Science 2.0 that have proven both promising and challenging at the same time:

1. The management of scientific data, both primary and secondary data (such as publication metadata, and other scientific content on the web) as a precondition for Science 2.0.
2. The recommendation of people and resources as a consequential next step in an exponentially growing scientific environment.
3. Quantitative and qualitative analysis of science based on data from scholarly communication on the web.
4. The change in scientific practices due to the involvement of Science 2.0 tools and technologies in the research process and the effects this has on science itself.

Topics of interest include but are not limited to:
* Definition of data schemes and interoperability formats
* Semantic Web standards for Science 2.0
* Social mining and metadata extraction in academic resources
* Metadata quality and quality assessment
* Design and architecture of data sharing facilities
* Systems design accounting for standardized data sets
* Applications for recommendation in science
* Specific challenges for recommendation in science
* Information retrieval in academic papers
* Recommendation algorithms and quality indicators
* Changes in scientific practices due to Web 2.0
* Methodological issues and interdisciplinarity in Science 2.0
* Opportunities and threats for researchers and research organizations
* Applications in and for Science 2.0
* Awareness-support for Science 2.0 activities
* Crowd-sourcing in science
* Robust methods for dealing with noisy crowd sourced data

Important Dates
—————
30 April 2012: Submission of full papers (8 pages) and demos (4 pages)
31 May 2012: Notification of acceptance
30 June 2012: Camera ready version (8 pages)
5 Sept.-7 Sept. 2012: i-KNOW 2012 Conference

Submission Procedure
——————–
We are inviting research papers of up to 8 pages including references and an optional appendix. Furthermore, we invite demos for the special track. Demo submissions should consist of a 4 page description that allows us to judge the quality of your demonstration. The Conference Proceedings of i-KNOW 2012 will be published by ACM ICPS.

Paper Submission Details: http://i-know.tugraz.at/i-science/paper-submission

In case of problems or questions concerning the submission of papers, please contact the track chairs at pkraker[at]know-center.at

Notification of Acceptance and Publishing
—————————————–
Authors of accepted papers will be notified by 31 May 2012. Accepted papers and demos will be included in the Conference Proceedings. The Conference Proceedings of i-KNOW 2012 will be published by ACM ICPS. At least one author of an accepted paper must register for i-KNOW 2012 before the deadline for camera ready versions (30 June 2012) in order to get the paper published in the conference proceedings.

Chairs of Science 2.0
———————
The organization team of the Science 2.0 Special Track consists of the following people:
*    Peter Kraker, Know-Center Graz (Austria)
*    Roman Kern, Know-Center Graz (Austria)
*    Kris Jack, Mendeley (UK)

Program Committee
—————–
*    Hendrik Drachsler, Open Universiteit Nederland (Netherlands)
*    Erik Duval, Katholieke Universiteit Leuven (Belgium)
*    Olivier Ferret, CEA Saclay Nano-INNOV (France)
*    Michael Granitzer, University of Passau (Germany)
*    Greg Grefenstette, Exalead (France)
*    Paul Groth, VU University of Amsterdam (Netherlands)
*    Denis Gillet, …cole Polytechnique FÈdÈrale de Lausanne (Switzerland)
*    Min-Yen Kan, National University of Singapore (Singapore)
*    Daniel Lemire, LICEF Research Center (Canada)
*    Jean-Louis LiÈvin, ideXlab (France)
*    Isabella Peters, Heinrich-Heine-Universit‰t D¸sseldorf (Germany)
*    Jason Priem, University of North Carolina (United States)
*    Wolfgang Reinhardt, University of Paderborn (Germany)
*    Katrin Weller, Heinrich-Heine-Universit‰t D¸sseldorf (Germany)
*    Fridolin Wild, The Open University (UK)