>>Welcome to Design 1 Session. Can I have your attention? We are going to start up with Peter Bryant on harnessing the power of the massive.>>Right. As it says there, my name is Peter Bryant– let me take this off, so I can work around. My name is Peter Bryant. I’m from the London School of Economics and Political Science in the United Kingdom and this presentation is along with my absent colleagues Chris Fryer and Darren Moon who we couldn’t afford to send over, so I got to come. So, for those of you who don’t know about the LSE, we are single faculty social science institution based in the center of London. And we have about 10 and a half thousand students. In the UK, we’re considered to be one of the sort of elite universities in the United Kingdom, and as such we had path throughout to our door over many, many years from people offering us alternatives to delivering our materials, some of our academics are world leading in their fields. We have people like Thomas Piketty, who was called one of the leading economists in world who work for us. And of course that content in that sense is worth a hell of a lot to other people. For us though, we looked at why we are a single faculty institution in social science, and what makes social science really interesting to us is the potential of social science to transform society and transform the world. And tying in with what George said this morning about the power of online learning and the power of MOOCs to transform society and the role of the humans in it. It was something that fascinated us as an institution but wasn’t really being delivered by any of the other sort of MOOC providers. So for us, we ask ourselves some simple questions and this actually our building just about to be demolished. Most modern pedagogies pretty scaffold sequential aligned and structured. In that sense, you go it from Week 1 to Week 2 to Week 3 to Week 4, you complete, you get some sort of certification, and there’s this idea that you need to build knowledge up through successive iterations of learning something sort of the whole idea of crawling before you walk. However what we also realize particularly in our world is that learning, living, sharing, acquiring, connecting and experiencing rarely are as scaffolded and structured as that. There’s this wonderful museum in Zagreb in Croatia called the Museum of Broken Relationships, which is all about a femoral items that are left behind when a relationship breaks up. And it’s such a snap point in time that isn’t particularly sequential, and you sort of have to work your way around it not knowing what happened before or after but it’s still trying to identify and understand the story. So we don’t live their lives in particularly sequential ways, yet most of the education we deliver is quite sequential. So we ask ourselves a question. What if learning could actually be informal, it could be community led, it could be non-linear, didn’t have to be Week 1, Week 2, Week 3, Week 4. It could be democratic as a process. It could solve problems rather than simply providing the skills by which people could solve other problems. It would be collaborative. It could be chaotic, I mean, may not actually be particularly ordered. It might be actually challenging and difficult for the learners inside it. But in that sense, it might aspirational, emancipatory and even open. So when we started, we asked ourselves these very simple questions. But then we also I just said well if we are important with our content is important and potentially society shaping, we want to do this in the massive scale. We have 10,000 students and that’s all we can take on our very cramped central London campus. We don’t have a particularly massive online provision. We have some public events that are well attended but they’re not necessarily for our students. So I want to talk about a project we worked on called Crowdsourcing in UK Constitution. It’s probably quite important locating myself as we are in the United States of explaining that the UK does not have a constitution, which is quite a– it’s actually a fact. We are one of about six countries in the world that don’t have a constitution, New Zealand, Saudi Arabia, Israel and Canada I think has a sort of semi-undefined constitution. It may not be officially declared as unwritten constitution. What we have is an 800-year-old document called Magna Carta which was the– 800th anniversary this year. We have then over 40 different active parliament and treaties and the like that form a constitution. So there’s been a belief from a number of people in our school, which is, why don’t we have written constitution? Most other countries do. What would a written constitution for the United Kingdom look like? And how do you construct one in a modern era when a lot of these documents are sometimes centuries old and based on a whole lot of actually quite boring and dry, dull constitutional law. But what we’re also interested in is that when these things are written, there wasn’t this wonderful thing called the internet. And all of this is now occurring in an environment of digital citizenship. And what we’re really interested is not just how you learnt to be a digital citizen nor what a digital citizen actually was which is the kind of persons of engaging in civic engagement through the internet. But we’re really interested in because you actually learn through being a digital citizen. I mean certainly, I’m– as you might tell, my accent isn’t English, my accent is Australian. We didn’t have the kind of civics education that the United States had but we sort of learnt the process of what it meant to be a citizen through action, through voting. We are one of the few countries in the world where voting is compulsory, so therefore everyone votes in Australia over the age of 18. Although that can be slightly misconstrued and say it’s actually compulsory to attend to vote not to actually vote itself. So what was this project? And that is King John who wrote Magna Carta using his laptop as he did. It was a three-stage project run by our institute of public affairs. They started doing it twice, once had about 40 people involved, the second platform had about 12 and was derailed by one person who answered every single question on the platform about– whether it would be about council ordinances of dogs or garbage collection or human rights with England needs a queen. And so, they shut the platform down through slight lack of interest. So they– for them, this wasn’t an educational project, this was a civic engagement project. They wanted to crowdsource the UK Constitution. They wanted all of the UK population to be involved and come up with this constitution. They gave us a very short period of time to write, make this thing live about 10 weeks. And what we did is we helped them take participants through the idea of the hacking phase. Well, you know, this is a very dry, dull constitutional thing, how do you get people involved and writing about it. We you stop them hacking why you have the constitution in the first place. We then got them to refine their ideas. And the idea or the platform was it they generated ideas about the clauses of the platform. The whole crowd generated it amongst ten different challenges. So we didn’t, so to say, come up with the constitution. We sort of said, “OK, look around human rights, around the head of the state, around a whole variety of other clauses, what things should be in our constitution?” So they came up with ideas, they refined those ideas as a crowd, they prepared them. And in the end, we had a big sort of both virtual and real event where they hacked act together a constitution. What we wanted to make sure was the entry and exits into this were open and flexible. There wasn’t a Week 1, there wasn’t a Week 2, you could come in at any point in time into the platform and stop to actually engage, and we run this project for 14 weeks. So what we decided to do is apply our online learning design to thinking about how we would deliver this outcome for them. So what we built, we took a combination of learning approaches, we use community leaning, we use informal learning, we use some of the process that come out with civic engagement around hacktivism. We even tried to leverage something that’s often used as a pejorative term and that being clicktivism. You know, participation can simply be a click away. We wanted to integrate a whole lot of participation practices into these. We wanted people to talk to each other, we wanted people to engage, we wanted people to be social but we wanted to do it in a massive scale. We didn’t want like a click– small communities to form. We wanted engaged individuals and groups, we didn’t want people just to sit back. We want people to participate, that’s what democracy is about. There were no readings in this. This wasn’t a course. So there was no formal documentation, there wasn’t an introductory lecture from a, you know, renowned professor saying, “Hello, I’m going to introduce you comes Constitutional Law 101.” This is a dry topic for a lot of people. So to have a lecture about why it’s constitution, what the history of constitution was at no real interest to our community. We didn’t have a lecture, we didn’t have a teacher. We did have a bit of guru, a guy called Professor Connie Gerety [assumed spelling] who was one of the world’s leading human rights academics. But his job is to go in there, was to point at people and say, “Now you said something”, that’s really smart. What if– point at someone else and say actually what you said, that was really dumb. Now let’s talk about why that doesn’t work, why can’t you just murder the Queen and have her hanging from a, you know, pike pole in the center of Buckingham Palace. There was no sequence to this. If you came in Week 1 or you came in Week 14, you could participate equally all the way through. What was really interesting, even though there wasn’t any expectation from our colleagues in the institute, there was an expectation of people who were in the course or in the program that was going to be learning going on here. Why? Because it was an institution delivering it. Matrix, I’m not that particularly fuzzed about. It was mainly to put a photo of baby Tembo who’s a statue at the LSE but, you know, we had that 1500 users, 725 ideas around clauses in the constitution, 125,000 idea views, 10,000 comments. The single most important thing for us is we wrote an 8500-thousand-word constitution for the United Kingdom. That was legally valid, slightly contradictory, had some really whack clauses but was presented to the speaker of the house, was mentioned by the current chief minister of Scotland as an example of democracy and a country who will be probably sometime in the next five years burgeoning democracy of their own. Our crowd produced a constitution which is what we really want– what we really wanted. What was fascinating is our community grew over time, it didn’t shrink. We see a lot of this matrix about, you know, x number of sign ups versus– then joined– we went up, we didn’t go down. We actually had 55% of our community are participating in the final week of the project. What was they doing? A lot of them were not adding new ideas. They were voting ideas up and down. They were participating in the democratic process. Eighty percent of our participants said that they’ve gained new knowledge by doing this. Seventy percent said they gained new skills. This is the question, I think I put a question up on Twitter about completions and how we define completions. We ask the question did you learn something. And this is what they told us. Eighty-eight percent of the participants were influenced by the discussions. Fifty percent stated that they were working with others directly contributed to their learning experience. Fifty percent changed their mind about civic engagement. The thing that we loved as an institutional, of course, is that 70% of participants believed the project changed their perceptions of the LSE. For anyone who understands the LSE, the LSE has a particular reputation in the United Kingdom for being as mostly conservative institution and very driven by where our graduates end up. That’s not where we started. We started training people to be administrators. We were very Phibian [assumed spelling] left wing institution. And for this kind of project, to change the view of the LSE was quite a positive thing. So I just want to briefly talk about three things on how this challenged a little bit of sense for us. For us, it was to define what we meant by massive. We weren’t concerned about massive by numbers. What we were concerned is what can a massive group of people do to change society. How do we leverage this massive to be more than just a number? How do we realize that the whole is greater than the sum of paths? These were not people participating as individuals. These were people participating as a community using a variety of methods to learn from each other, from their own desire to learn, from challenging others, from questioning others, from debating others. And for us, it was a process of redefining what the massive actually means, as not simply a metric of number or representation or activity but a massive around common experience. This was in some way a little bit of a cheap shot between when you talk about a community of learner as a learning community because there isn’t really particularly differences between them. But what we were looking at is how do you extend the community practice, so this people who all have something in common, interesting constitution, how do you extend that to be a learning community? Sorry. How do you deal with community formation and citizenship? How do you do with people owning that community, not feeling that they were just a, you know, a cog and a wheel, that they were actually a fundamental part of the community. How would they were rewarded for participating the community? There’s no certificate, there was no badges, it was a slight gamification aspect but generally what would I rewarded for? How do you make this– how do you keep momentum over 14 weeks? How do you keep it sustainable? How do you keep people engaged? How do you get 55% of people participating in the last week? And how do you deal with the fact that you’ve got coming into this a whole lot of variable learning trajectories schema and pathways? We didn’t ask for a lot of information. This wasn’t a traditional cause. People came in for a variety of reasons. One of the probably most important things we did was engage in all of these special interest groups who were interested in this, the monarchists, the republicans. And we said to them, don’t try and destroy this process, whether you like or not, come in and write something, tell us what you think, bring your members along. And what was great was Liberty which is one of the big republican groups in the United Kingdom. They brought in nearly 500 people. We thought they just spend up talking about, you know, bringing the Queen down but they didn’t. They filtered through the rest of the community and engaging those groups. And not engaging individuals, they’re engaging groups. And we also wanted to contest what it meant to be open. Because we had no beginning or ending, open is something that is particularly important to contest. The notion of digital citizenship is something that is equally open. Opening up the academy and what it means because the LSE is, I said, quite a privileged institution, it’s quite hard to get into. This is something we just considered to be part of our activity to open it up. We wanted to embrace nonlinearity and we really wanted to talk about open in its truest sense. So, it wasn’t that you once signed up, you got access to it. You could look at anything on the platform. You could look at any of the debates, any of the comments, any of the process without signing up. But the moment you signed up, you started putting your name to things and that created a very different relationship. So I just want to leave you with three simple questions. What happens when you empower a community, like this, to learn and engage in social change? This is a project that is very rooted in social science and very rooted in digital citizenship. But can it be more than civic engagement? Could you use this model of people activity engaged as community in a nonlinear way to solve particular problems? We have this wonder example that we have piloting as our second test which is around climate change. We’re getting people to send in photos of their own local areas that have been affected by climate change. What we will have, we hope, is a collection of tens of thousands of photos of climate change. I hate to bring a data set in fear in academic, that’s like matter from heaven. But more importantly, you’ve got this wonderful collective computing process to look at all those photos, anyone who maybe around my age, remember the old city project? Anyone has heard of the city project, where they use to let data from distant satellites process in the computer in the background. This is kind of a human city project. And does this help to build a more informed digital citizenry? A group of people who are informed not just about what digital citizenship is or what social science can do but actually act and use that through their own processes. So I’ll leave it there, thank you. [ Applause ]>>Thank you. So we have time for maybe two questions. And while we’re taking those, is Nina here for a presentation?>>No, it’s us presenting.>>OK, yeah. Well, if you want to come up and load up your presentation while we take some questions. So here you go. All right, I’m going to hand the microphone to anyone who has a question.>>That was great talk, thank you very much. Just, if I could ask you to make maybe one or two more steps that I think your presentation gesture towards, do you see kind of like a collective project like this, is it– would it be possible to apply this type of method to more traditional MOOCs too and I wonder if we would see similar results. Like if we had a collective project in a physics class or a chemistry class, do you think we would see this type of participation or attention?>>I mean, I guess for us because it is very much based in social science that I would be guessing. But my guess would be we know that social learning is an effective model of learning across the disciplines and across the age groups. We know that even in more traditional areas collaborative work with students, works. We also know that’s what they usually end up doing in their employment as well. But, you know, you take the example I’ve talked about around the climate change thing. So, you could take that same data set, and instead of looking at from a policy perspective, you could actually start to look at it from a geography perspective. You could start to look at it from a kind of a science perspective. And you use that same information, that same data set to deliver exactly the same kind of outcome. What we did workout with this is that this is very specific to a purpose. All education needs to have a purpose. All pedagogy needs to have a purpose, the driving tools. The purpose for this is we have a problem to solve. We have an outcome to deliver. If it’s simply a dissemination of information project, then I don’t think it would necessarily work.>>Thank you very much.>>OK. One more question? You can get started?>>Yup.>>Yes. OK.>>I guess–>>Thank you sir. Great. [ Applause ] OK. So, the next presentation is Pedagogical and Technological Concepts for Collaborative Learning in MOOCs by Julia and Tobias.>>Yes. Thank you. So, my name is Julia Erdmann and– thank you, this is Tobias Hacking, we are from Ruhr University in Bochum and University of Duisburg-Essen in Germany. And we are here today to give a talk about Pedagogical and Technological Concepts for Collaborative Learning in MOOCs. Project is founded by the Mercartor Research Center Ruhr, where you probably don’t know.>>Can you please use the mic?>>Thank you. This morning George Siemens raised the question of how we can make education more humane or how to better connect people in online education and make it more social. To bring small group collaboration into MOOCs might be one way to facilitate social factors such as social relatedness. As we know from decades of research on collaborations– model collaboration can facilitate learning on different levels. Besides social relatedness it can foster motivation and of course it can help students on a cognitive level by, for example, sharing or exchanging information. And a small group collaboration is a– however, not well, sorry– It’s not a well elaborated subject yet in MOOCs, so this is why on one side we want to investigate the impact that’s more collaboration as more of a collaboration can have on learning with MOOCs. And on the other side if we turn it around, we can also see the great potential of MOOCs that research on small group collaboration can profit from. The diversity and high number of participants for example has great potential for deriving data from learning analytics methods. And this is where the loop closes because these analytics could then form the basis for forming and supporting groups in new ways. Before I introduce you to our project, I will briefly position our project in the realm of MOOCs. The course we are running is not a MOOC in particular but a MOOC-like course. We locate our course somewhere in between the concept of a MOOC as offered by Coursera [assumed spelling] or ethics. And that of I spoke which is a small private online course most of you probably know the concept. The course was open for two large universities which is Ruhr University of Bochum and University of Duisburg-Essen. The universities both have more than 40,000 students each. However, the course was not open for general public and it was not massive as– in our first run it didn’t have– not thousands but hundreds of participants. It is bigger than a typical spoke though and more open as many students from different study programs from two universities can participate. Another difference is that differs from a MOOC that the course credit was relevant for the participant study programs. Like in a MOOC, of course, comprises of purely online learning activities that incorporate typical style elements of a MOOC such as video lectures and automatically evaluated assignments as well as discussion followups as communication tools. So, back to our project, to achieve our goal which is to bring effective small group collaboration in, I will now call it online courses not MOOCs. We are lucky to have a perhaps especial consortium because it comprises of researches from very diverse research place. This diversity allows us to elaborate our research topic from a very diverse perspective. We have in our consortium expertise on Educational Psychology with a background especially in computer supported corroborative learning. And we have expertise in Social Psychology with a focus on the use of new media and communication and we have expertise on learning and knowledge technologies based on learning analytics. With these competences, we are working on the following questions. First, how do we need to compose and structure groups to make group work MOOCs efficient. Second, how do we get students motivated to make group work and MOOCs efficient. And third, how does the instructional material need to look like to make the group works and MOOCs efficient? So in this talk we are going to focus on the study one which is about group formation so let’s first of all have a look at the structure of the group assignments and the overall appearance of the course. So, here on the left hand side you can see organizational elements like a calendar and there space for assignments as well as news and the middle you can see the ringing that tips which represent the topics or the lessons. And when you unfold them you can see the materials and activities such as the video and literature and we have the quiz and we have additional or optional literature and the subject that is central for research, the group assignment for this week provide two different work environments. We have a creative discussion forum where students can coordinate their group work and ask questions of any kind and then we have a realtime text editor or where students can simultaneously edit their text or their product.>>OK, so I’ll take over. So as Julia already said the major concern of our project is on learning groups and smaller groups and learning groups in MOOCs especially with regards to group formation. Yeah, group formation is not a new topic. It has a long tradition but it is still of interest in pedagogy. However, in MOOCs, things get a bit complicated since we have a high number of participants manual assignment of learners to groups is simply intractable. So, but on the other hand that we gather lots of learner generated data through the platform, for instance we saw a huge [inaudible] contribution for discussions to it’s resides and so on. In our approach, we call it analytical group formation is to make used of this data to form heterogenous groups of students with diverse characteristics and homogeneous groups was students with similar characteristics. Our assumption is that heterogeneity is beneficial for group learning in general. And we can also use the large amount of data and to diversity also participants in MOOCs to establish heterogeneity. So, we evaluated this approach in two small group tasks whereas subject groups each of course students. In group task one we formed heterogenous and homogenous groups based on resource usage in student coverage of optional and mandatory literature, coverage of videos and attempted quizzes and the second group task we used data gathered through group task one to form groups based on activities in the group discussion forum. So we’ll skip the next slides since we’re running a bit out of time and go directly through the [inaudible] files. We have some initial findings as I have said we have a group task one, students used collaborative different item and the discussion forum that we used to activity data from the discussion forum namely the number of characters supposed in the discussion forum the text quantity to establish homogenous and heterogenous groups for group task two. And what we find in homogenous groups was all students have a high level of previous group activity. They are the most active groups but yeah they far have offset the homogenous average and homogenous lower groups but surprisingly we see that heterogenous groups also above average and lower level groups so that we can conclude that forming heterogenous groups might increase the overall activity of the cause. One we now compare students was in a class namely high average low. We see that high level students post a similar quantity of text to the discussion forums in homogenous into heterogenous groups, so heterogenous groups are slightly of it and heterogenous group they post slightly more on average but it’s not significant. In contrast to that, in the low level students posted significantly more when they were assembled in homogenous group with other low level students. And so they are far from the text quantity of the high level students but for the lower classes, lower level students in homogenous groups they were much more active. One possible explanation could be that we– you know, we have to face something like social loafing. So a low level student in the heterogenous group can rely on the high level and average level of students to do the job. But in group with low level students, they have to become active to some extent in order to solve the group task. And second finding law state in most of the groups we have at least one inactive member. Inactive means has not posted through the group forum or not participated in the collaborative writing activity. And the question now is we aim to answer in the upcoming months is how do groups deal with inactive members. Do they suffer from inactivity or can they compensate as certain number of inactive members, so this is something we need to find out. Research finding– we have asked in chroma’s collaboration we you have no face to face activities and to post there’s a groups discussion forums and activity in the collaborative writing to where is the only way when the only channel in which it becomes visible to other group members that team mates are active that they’ working on the assignment. And we have here two examples for activity trajectory so each lane corresponds to up the activity of a particular student over time so blue circuits mean forum post, rectangle means, that it has working something of the collaborative writing and we often find groups was on lot activity gaps. In such groups we often discover forum post like who are you? And are you still listening? Or we also received e-mails where students ask to whether they have technical problems since their group members [inaudible] team mates did a post sends to the group forum and this is really a major challenge for group learning and MOOCs to overcome this deficit of group awareness. On the other hand we have– we’re coordinated groups where we see that there is direct and immediate response to each foreign post and we also see some kind of periodic pattern of active periods and inactive periods. So these groups are well-coordinated and much more productive than the other groups. So last slide, so based on this initial finding we could identify three future challenges. First, we need to create better condition for effective group work. This Could be for instance the root formation of fruits if the number of inactive members becomes too high, time scheduling to overcomes this– yeah, let say issue with group coordination when students have different times when they work for the course. And we need future [inaudible] for collaboration. Second challenge is activation. So we have some drop outs so they will never be active at all but they are potential active participants and we need mechanisms to activate them to participate in group works. This could be for instance, incentive elements like bachelors or scaffolding in form of active prompting. And so our challenge, we need to enhance the accountability. This is good for example, be achieved through peer review and feedback so that as a student feel a bit more responsible for the outcome of the group work and we can also try to overcome the social loafing effect by different group compositions. So that is all. Thank you for the attention. [ Applause ]>>Thank you, thank you. So we have time for maybe one or two questions and Drew if you want to set up. I’ll bring the mic over for your question.>>Thank you both for your talk. My question is around– I was really interested in your conversation about heterogeneity in groups and I was wondering if that is particularly relevant to the subject matter X– the subject matter that’s being discussed. This is sort of like one of those holy words that you get into when you start talking about group work and one of the things that I’ve heard a lot is that it depends on the course type. So I’m wondering if that’s something that you thought about or have thoughts on.>>Yes, we have talked about it. And this is a really good question. Yeah, you’re right. There are group assignments that are much more suitable for group work like assignments that require discussions among the participant that that require exchange of opinions and knowledge. These kinds of assignments are much more suitable but there are– of course, there are assignments that can be better solved by individuals, the same as for causes. If we have for instance– yeah, let’s say an art class or a muss class. So there are different types of assignments in such causes that are more suitable or less suitable for group work. So this is– But for sure this is something we consider on the pedagogical level in our project.>>Anyone else have an– no more question? OK, great. Thank you. So our next presentation is credentialing MOOCs in an open degree by Drew Ross.>>Hi guys. This is going to be probably really hard for be because I’m kind of a walker and being stock here behind the podium is going to be a little difficult but I’m going to give it a try. So by the way just in case you don’t catch it later, my e-mail address is the easiest e-mail address in the world to remember. It’s just my name @kaplan.edu. So if you want to get in touch please do. This is going to be a total change of pace from what we’ve been talking about. I’m actually going to be talking about kind the administrative and structural functions around a degree that we’ve created that allow students to use MOOC learning and OER learning to earn a degree. So without further ado, let me tell you a little bit about what open college is. Open college is a brand new division of Kaplan University. Kaplan University as you guys probably know was owned until very recently by the Washington Post Company. Kaplan Inc. was owned by the Washington Post Company. And so we have kind of– for a very long time had this long tradition of trying to do things a little bit differently from the way a lot of our competitors have done them. And one of the things we’ve done recently is we said, all right, you know, we’ve got this degree and it’s– not to degree I’m about to tell you but we’ve got all this other degrees at the university, and they’re– you know, relatively high cost. How do we turn things around create a very low cost degree for students that allows them to have a lot of flexibility and allows them to do essentially what we know from feedback from students they want to do. So in 2014, the open college was created and was given three major tasks. One was to handle all the University’s prior learning assessment. So at open college we do all the assessment of experiential learning, work base learning, volunteer learning, that sort of thing. We also have open courses. We started our first open courses in 2012 and we have– I think at this point eight of them that are running. We have 12 of them that are theoretically active but only eight of them are running right at this moment. And they really run the gamma. They’re all base on original Kaplan University courses and they are for the things that you would expect they would be about. They– We have one on one communications, we have one on basic introductory composition. We have a math course, things like that, sort of very sort of general end courses. And then last year, the university started an open degree at Bachelor of Science in Professional Studies and I wish BSPS have not been taken by paralegal studies but it has. So we are BSPr and not BSPS. The open degree looks kind of like nothing that most people have seen before and then we’ll have echoes of things that people will see it before. The idea behind the Bachelor of Science in Professional Studies is that it is a generalist program. There is really no specific focus area to the degree. It’s kind of an interdisciplinary degree that allows a student to focus on whatever area he or she need for a career. And we quoted “distributed competency based education program” because we actually don’t deliver any of the curriculum. The curriculum comes from other people. Curriculum probably comes from you guys and we’re not the actual deliverers of that with the exception of one class and that’s the [inaudible] of course that we’ve created. We have a couple of other kind of– they’re not actual courses. They are like tutorials that we’ve got for teaching students professional competencies, things like team work and leadership, working with– and multicultural and diverse environments. And then there’s also one about professional presence. So we do teach many courses on those things but generally, all the courses students take from general chemistry to advance accounting comes from elsewhere. So here is the structure of how the program works. Here’s the sequence. A student starts out with us. They enroll and then we do a prior learning audit for them. We work with them to determine how much credit they should get from other sources from their previous learning from transfer. We then have a faculty member work with the student to develop an individualized learning plan that really does speak to what the student wants to do with his or her degree. And then here’s where things go a little different and this is where– and a lot of people ask a lot of questions. So, you know, feel free to come back and ask me questions about this later. What we do is we take Kaplan University course outcomes and we match them up with very broad categories. So in the ILP, the student won’t see, you need to take Biology 103. They will see, you need to take 100 level Biology course. And so what we’ll do is we’ll give them the Kaplan University catalogue and we’ll say, “OK, do any of these 100 level Biology courses look interesting to you.” And the student almost always says, “OK, yeah, yeah, that one looks really interesting to me.” And so what we do is we give the student the outcomes. We say, “OK, that course has these five outcomes.” And then we work with the student to find OER MOOCs, other open education resources that will allow the student to master those outcomes. We have a proprietary matching tool that we created that actually if you enter in the names of the outcomes, give you internal language. It actually matches up with MOOCs and OER connected up through things that you guys probably already know about things like OER common, that sort of thing. And we say, OK, well, for outcomes one and three this MOOC might help you. For outcome four, this piece of OER might help you and for two and five, we’ve got this other MOOC or here’s another MOOC that might help you. And we also do have some of our own open courses so in the situations where ours match up we always pair them with our first. And then the student goes off and this is a self-paced program also. The student can take however match time he or she would like to complete his or her learning. When the student feels ready come back, the student comes back and we do an assessment of learning. And I’ll come back to that. Once the student has filled in all the slots on the degree plan, the student does a capstone course and the capstone course assesses all the program level outcomes for the BSPr and then the student graduates. So, I’ve kind of already gone through this a little bit but the interesting thing about this that is a little bit different from other models that you’ve seen that look like this is that we do have our own special tool that goes through and scrapes all the information that we can find. It’s an API based program that does a whole lot of scraping and grabs information about all the MOOCs, all the OER that can possibly get its hands on. And then when we have a conversation with the student, we use that information to identify resources that might match up well. So here’s how we do the assessing. This is always a question people want to know about. We– In some situations have challenge exams. So we’ll have an exam for that Biology 103 class. And so we can just give the student the exam and say, all right, you know, show us that you know this stuff. But a lot of our courses don’t have challenge exams. So we needed a method to figure out how to asses that learning. So what we did is we created something called the PANel. And a PANel is very similar to a very qualitative evidence-based challenge exam. All it is, is it’s a portfolio of new learning that’s where you get the P-A-N little e-l. We like little letters in our abbreviations, don’t we? So what the PANel does is it gives us a chance to say, here’s a list of eight pieces of evidence that a student who has mastered all those biology 103 outcomes will be able to supply us. We want you to choose five of these, half of those have to be writing intensive elements. So we want to make sure that we’re assessing writing throughout the curriculum as well. And you’ve got two weeks to do it, knock yourself out. But not really knock yourself out. And the students really do take to this very, very well. This is an environment that they really kind of thrive in and we’ve had, I think 11 student so far go through the PENel process for, I think three classes each and they are all– every single one of them has passed. All these assessments are pass/fail and we always make sure that on the evidence creation list side and also on the evaluation side that we have a faculty subject matter expert. So what we’ve done recently I we went to our students and we talk to them a little bit about their experience with the BSPr. Now, as you can imagine the BSPr is an experience that is pretty much unlike anything that students will have seen before. So we wanted to find out how well they were taking to this and what kinds of difficulties they had. What kinds of things they saw is important to their success. So, we did a very, very long study that involve lots of interviews with our students. We interviewed almost everybody and we had an approximately 45 minute conversation with each of our students asking them the only five questions, five very open ended questions and they really, really like to talk about their experiences. Here’s what we found. Students almost universally identified advisor support and faculty support as the most critical thing to their success. If they didn’t have faculty support, they couldn’t make it through and we know that that’s true even though we’re not actually delivering courses to them. We’re not actually teaching them the content. They still said we need the conversations with the faculty members. This is the faculty members, remember, that created the ILP with them and then also the faculty member guides them to resources that will help them master their learning. Students across the board said that they loved driving their own learning. They kept using their language, driving my own learning. I love that I’m able to drive everything. So that refers both the pace and it also refers to choice. Students really were very clear that this was not anything that they had experience before and they really liked that. This was– In almost every single one of the cases our students have a significant amount of previous college experience. So these are not brand new learns. These are not kids running on high school. These are people in their– generally their 30s, 40s and 50s. And they all identify this as being something that was unique and something that was really beneficial. And then the last figure is that they really appreciated the transparency of the program. And it made them care more. So this gets of something we heard about earlier about identifying what was going on making the students to learn– sorry, the students to teachers and the teachers to learners. If you think about this, if you’re able to identify what you need to do and why you need to do it and this really relates why you need to do what you’re doing, I think it really does help increase motivation. And students really identify that their motivation for this was based largely on the fact that they were in complete control. So that is it. I’m going to stop and I’ll let you ask questions. And I’ve just got my three minute warning about a minute ago, so it should be good timing. [ Inaudible Remark ] Just give him the microphone. Sorry.>>It’s all right. Rob Steiner, American Museum of National History. How would you compare your program to the competency-based program at Western Governors University and you regard them as direct competitors of yours.>>It’s a good question. First of all, ours is self-paced which makes a big, big difference. The Western Governors program is not really a self-paced program. Another big, big difference is that our students are in more contact with faculty members. And they do have contact with subject matter experts if they needed. The faculty and the Western Governors program are– essentially they’re sort of generalist mentors. So, if a student of ours is going to go into a biology course and needs to talk to a subject matter expert, we both have the resources and we will very willingly, you know, put them in contact with that person and, you know, get them the help they need. That’s another big difference, but I think the pacing is a big and very important difference. You know, [inaudible] is doing something that’s kind of similar, but again it’s not self-paced. And that’s one of the things students identify that’s important to them.>>Putting this into a practice aspect obviously because learning, you know, as long as learning occurs in practice, there’s a lot of programs in the United Kingdom over the last 15, 20 years sort of used web-based learning as a sort of measure of assessing prior learning through work. Through some of your measures, do you build in where people are acquired skills through practice?>>Well, generally what we do is when we award credit for prior learning it has to match up pretty cleanly with something that we also teach. It’s very hard for us to engage if for example, you know, culinary skills because we don’t teach any culinary programs. So, there isn’t a way for them to do that. So to some extent some work-based learning is always going to be a lost especially the stuff that is very kind of place-based and, you know, very specific to particular context. But, the more general skills, you know, we’re pretty good at assessing, at least I hope we are.>>Hi, David [inaudible]. I’m very interested in model. Thanks for sharing. You said it was quite a lot cheaper for students, what solve level of fees? I’m really interested.>>Sure. The cost of our regular four year degree is somewhere around $50,000 total all in. The cost for this degree is somewhere around 11. It can be even less than 10 depending on the student and how much credit he or she brings in. So the fees are a monthly fee that a student pays for access to faculty and to the advisers. And that’s $200 a month and then the students pay per assessment. So, every assessment they take is $100 in addition to that. They also pay for the capstone course. They pay regular tuition for that or slightly discounted tuition. And that’s somewhere around $2000.>>I think we have time for one more question.>>Alice Kessler-Harris, Columbia University. Could you– I’m afraid this is a rather skeptical question. But, could you tell us what– how the quality of the education that you’re providing would compare with the quality of an education that a similarly situated university. I’m not thinking of Columbia but I’m thinking of an Empire State College for example which does similar assessment-based learnings. And related to that question why do you assume that if a student makes choices that those choices are necessarily go– they may produce greatest satisfaction as you indicate, but they may not produce the kind of learning that we would normally expect for a college degree.>>Sure. I don’t think that I’m privileging student choice over results. What we’re saying is that students appreciate this and not so much that that’s– meaning that they get better learning or worst learning.>>So are you charging students too because they appreciate what you’re doing or because you’re actually training them–>>No.>>– to do something?>>I think your original question was how to compare the learning. And it really depends on the resources they’re using. We’re guiding them to resources. We’re helping them when they have difficulty with whatever they’re studying. And the difference between what we’re doing and what somebody like Empire State is doing is if Empire State is delivering the content. Let’s just say they have– and just like we do they have, you know, like 10 week program in biology let’s say introductory biology. If the student goes out and has five outcomes for five credit, five quarter credit biology course and the student has access to all the OER in the world and the student spends the entire 10 weeks or spend 6 weeks or whatever using all those resources, resources that match the students’ particular needs, I would venture that a student is really motivated and, you know, efficacious and really understands his or her own modes of learning would be better off in a situation like that than going through sort of a generic program that was just created for a random learner who happens to be, you know, 27 and has a background of you know, X, Y or Z. So, it allows a student to really match up his or her own styles with what’s out there. And there’s a lot out there. I mean there’s a tremendous amount out there. So really when you ask about the quality, it’s not the quality of what we deliver, it’s the quality of what the world is delivering at some level. Now, we have to make sure that our assessments reflect something meaningful. And our assessments for these courses match up with our assessments for our regular version to the courses. So, what we’re assessing there is the same as what we are assessing with this degree. And those are assessments that, you know, have been through our regional accreditor and, you know, through Department of Education. So, we are always going to be responsible for quality at that level. And I think that’s a really good thing. I wouldn’t ever want to this and have someone to say, “Well, it’s just, you know, rubber stamp it.” But, it’s actually meaningful. And we are held accountable to the meaningfulness of it every four years or so.>>Thank you, Jerry.>>Yep. No problem.>>So, our next presenter is Piotr Mitros, the pedagogy behind Open edX.>>Hello, I’m Piotr Mitros. And– sorry. I’m going to use video. And this is the pedagogy behind the Open Edx platform and behind the MITx platform that had led into it. This is slight a bit of a historical detour. When we first created the platform our goal was to create a platform for exploration of different styles of teaching and learning. And we try to make a platform modular both in terms of how the content is designed and how the technology and science so we could do that. But when we want for first courses, we needed to design them in some way. And there was also very distinct pedagogy to this. To date, we really haven’t talked about it very much because we hope to foster communities like this one learning its scale and give the community the flexibility to build up the kind of innovation we see here. But we thought this would be a good timing to go back and talk about the design principles in our first courses as well as the technicalities and support. So, where we started was for the set of principles from learning that I think these represents all the strongest principles we’ve come across, Mastery Learning, introduced by Bloom [inaudible] until they’ve mastered the concept, a lot of follow up work, a lot of disagreement about what the learning games are but universal agreement that they’re large. Authentic assessment which I would say a lot of people believed in, but the evidence there is– if I were to create a platform, it was a little bit weaker. And so it’s a little stronger now. Immediate feedback/continuous formative assessment letting students monitor what they know and what they don’t know. Interactive engagement pedagogy, that basically means having students actively doing things rather than listening to a talk like this one. Constructive learning where if possible having students derive information for themselves for educators that travels is different from constructive learning which was not something we try to do at the time. Personalization in the way that [inaudible] would use the term, and that means having an emotional personal connection to the instructor that’s something can popularize. Remediation of misconceptions are books and conferences on this topic and conference to the other ones, the research evidence for this is– not existed or something that tells you towards systems community, that’s only weaker. And self-regulated learning which we felt was very important given our very first audience. But again that’s a place where there’s mixed research support. If you ever look at something like [inaudible] they will give a very nice rebuttal to this. So how do we build these? Oh, actually before I go how we build this, it seemed to work. When you’re at your first MOOC, 96% of alumni thought that was as good as or better than the residential course, at least for mostly college graduates. We’ve used that course in blended settings at this point. And we’ve also had tremendous improvements. Our first randomized controlled trial was at San Jose State University where completion rates went from 59% to 91%. We’ve had similar results in less formal trials of other courses at MIT. And one of our physics courses taught by David Precheur was evaluated relatively [inaudible] in MOOC setting where the pure online format was better than a traditional residential course although not as good as a blended course. So we have a lot of validation that is at least seems to work. And this has– this pedagogy had a couple of elements to it. One is that it was based on authentic assessments. So when we launched our first course, we in fact did not support multiple choice questions to the platform. The questions were old things like design a circuit; write an equation, give a numeric answer. Things that are sufficiently open ended that there’s an infinite set of answers. And what that allows us to do is first of all apply mastery learning or assessments, students can keep trying until they get it. They can monitor whether or not they understand something simply by whether they’ve gotten it. And most of these assessments, we have of data talk about three or four hours per problem to do. So these were complex analysis and design problems. It also– and that portion of the course routes we’re actively doing something. That’s the interacting engagement pedagogy. The [inaudible] versus learning sequences which are interleaved videos, assessments, pieces of text, interactives. And most of the videos are very short. Most of them were I would say, under five minutes long. In many cases, they were maybe 30 or 60 seconds long. And so students as they were doing these videos and going through the assessments could continuously monitor their knowledge, this was an active learning process. And when the school is well done we would have students actually derive information for them selves. If it was a mathematical proof, we would ask them questions walk them through how we do it, but actually have them do that work themselves. Now, we didn’t always listen very well. We were mostly active rather than constructive, but we occasionally did. And the video spoke when you saw before this one whereas I kind of replicate the field of two people sitting next to each other while tutoring another one either on the tablet or in a coffee stand or similar, very personal, very connection. This was popularized by [inaudible]. We didn’t invent this, but we did adopt this. So, again, that’s a place where we see interactive engagement pedagogy, a bit of constructive learning and a personal connection. In our first course, we started with a Q&A forum. Later on we moved to a discussion forums embedded right below every problem and video. That was there for the purpose of remediation of misconceptions. We also had ways for students to easily go through the course text. We had ways for students to easily navigate back if they missed something. All [inaudible] student phased out for a minute or didn’t understand something, there were additional resources to help them. The Q&A forum in particular worked very well. Ninety-two percent of questions were answered on our first course of those that quality of answers was very high and the media [inaudible] was about 12 minutes. And in most cases, the reason why this works so well is because students wouldn’t ask questions. They would look for questions that it’s already asked. Students read three times or try as many threads as they create it. So we had and continued have good remediation and since then we’ve added adoptive hinting that was worked from Stanford OLI. We added a recommender system that was contributed by MIT both coded out with ethics. So that gives us remediation of misconceptions. Self regulated learning more [inaudible] one time but that was primarily these navigational elements where students could skip around easily on a video, skip around easily in learning sequence, all those that bounce, I told you what those were as well as figure out where they needed to go through integrated assessments. So a lot of students would go in, do the assessments. If they got them right, move on. If they didn’t, go back and learn the material. So that’s a core of the problem. That’s a core of the platform. As we’re moving on we’re adding a lot more work in teaching and learning that you’ll see coming out in the next few months and years hopefully, some of these are edX projects, some of these are projects like research groups around edX. And there, I open up for questions. [ Inaudible Remark ]>>Sorry, it’s [inaudible]. You’ve mentioned where you’re going with that. So, when you say you’re doing active research to look at interaction patterns in a way that knowledge flows across that? What are you actually doing with these features that you have?>>OK. So teams, entity, project, I think Leslie could talk more about, but that’s being build into the platform right now. We’re sorry to dip our toes into adaptive. So there was a few collaborations with primarily actually driven by external Open edX contributors throughout adaptive. Now, we’re working to try to bring those in house. That’s at fairly early stages. Cohort is ways to group students for example by level that is at the platform right now. Peer instruction is being done by an external research group. It’s been piloted in one or two courses. Right now, we’re trying to figure out whether it’s something you can bring in to the platform. Peer feedback is being developed, codeveloped with Cornell. There is a small project around that. We don’t know which of these will succeed. These are research collaborations. Crowd source hinting is actually being developed in house right now pretty actively. We’re allowed to pilot with that a year ago. We asked students to contribute hand space simply by in house and tutoring system. We’re bringing that back into the platform right now, recommended resources was codeveloped with MIT. Students can say, “Gosh, just help me with this problem over this video and other students can see that.”>>Piotr, we’ve had this conversation before I think. And, you know, I’m a big fan of Open edX, but I want to call you on the– I use of the term self-regulated learning. Meaning, that the learner simply has flexibility to choose and click through which objects they’re viewing or interacting with. Of course, Open edX provides that. And that interactivity is something I appreciate. And it’s a very good feature. But self-regulated learning involves something that is happening to the learner when they actually have a level of metacognition when they actually can recognize the weaknesses and strengths and develop strategies for problem solving. So, I think you’re kind of co-opting the term for describing the platform when it really is not what we mean by self-regulated learning. We’ve had this conversation before last year, remember?>>We did. And I think I certainly agreed with you impart and disagreed with you impart. So the problem with actually all of these terms, let me just go back here, is that they mean very different things to different people. And if you sit within a particular research community, they will have a definition. And I agree that the definition of this as well as actually many of the– many other of these terms is incorrect within some contexts. I can also find context where these definitions is how specific groups of researchers use them. The hardest one here, I took out personalization, put in personal connection because that’s how it worked for [inaudible] personalization. And the intelligent tutoring people get very upset with that use of personalization. Constructive learning is hard because there’s no alternative and the constructivist get very upset at this term. And I can keep notes for actually all of these terms.>>Yeah.>>And so, it is a question of terminology. I tried to explain what I mean by it, but it’s certainly the case that a lot of these terms are misleading depending if any other you’re a cognitive scientist sort of will teaching back on those truffle desks, design background, educational philosophy, physics education research and so on. It’s hard to please all of these groups. So, I apologize. I am misusing this terminology badly within many communities. [ Applause and Laughter ]>>Very good. [ Applause ]>>Is there any community that use this? But, is there any community that uses the terms self regulated learning to mean something different than what I just described?>>I’ve heard– used the way that I am using it.>>By somebody other than you and edX.>>Correct. I am happy to Google around and figure out which group is at this way, but I have heard this term.>>I will challenge you in every conference with the same question.>>That’s fair. I will come better prepared and figure out which communities use it before the next conference.>>OK. This is a good debate to carry on to the reception which is starting at 5:30. It’s in the Everett Lounge which is Zankel. It’s that way. I believe that there’s going to be some hall directors that help you get there.>>Marshalls will direct you.>>Marshall should be directing you out there, but it’s over in Zankel Building across the way. Thank you.