[00:00:07]
>> He's also. I am. Thank you. So yeah I'm really thrilled I came down and what Georgia Tech is doing with the year scalable advance learning environment I think is really at the forefront of thinking in higher ed right now and so it was really impressive to me and I'd love to talk a little bit about some work that we're doing but also engage with you in how you're thinking about where you want to be going because I think in some ways you're sort of going through uncharted territory and it would be great to be collaborating a little bit more on some of these things.

[00:00:51]
So a couple quick attributions Rob Abel is C.E.O. of I.M.S. global Robin I work very closely on the chair of the board right now for I.M.S. global and I've been that way for since 2015 and so a lot of the vision of I.M.S. is really Rob and on here helping to just sort of share that Vince Kellen who is the CIA At the University of California San Diego Vince is also on the board of I.M.S. global Vince and I have really been spending a lot of time talking about next generation analytics environments and so one of the things I'm going to talk about all during my talk is sort of where we're looking at going in creating a next generation analytics environment to be able to take advantage of all the data that learning analytics can actually provide And finally I want to thank Malcolm Brown who is the director for the Edge a cause learning initiative Malcolm has been one of the people who has really been preeminent in thinking about learning and the whole next generation digital learning environment comes out of work that Malcolm is done as part of the gates grant that was there and so be glad to sort of fill in how this is all connected to that as we talk later.

[00:02:09]
A couple things about you N.B.C. that you know it's hard to pronounce the name we turn out we're the only research university that somehow has the word County in our name so you N.B.C. is 15000 students we are 52 years old found it in 66 we are just outside of Baltimore Maryland if you've ever flown into B.W.I. Airport we are about 4 miles from B.W.I. Airport we are about 55 percent of our incoming freshman are stem we predominantly focus on engineering I T. in the sciences but we have a number of other.

[00:02:51]
Majors which are very digital media and communication study is very digital are in and we have an animation an art program which is very digital so it's a very I.T. centric what's interesting about you N.B.C. is we started a program back in the late eighty's we now lead the country in sending African-Americans or on to get Ph D.'s in STEM and that's been a big thing for us because when you look for a small school the size of the sea many of those students have actually come to Georgia Tech got their Ph D.'s here that are there but we see that as one of the ways that we are thinking about how we innovate in learning so the idea of starting this program back in the late eighty's that required us to sort of rethink the way we do education how we bring people in how we have them work together in teens and so we've had this sort of focus for the last 25 years in thinking about how we move forward as an institution in the teaching and learning space.

[00:03:58]
For the last 10 or 12 years if you ever look at U.S. News and World Report you have B.C. tends to show up in the top 10 for best undergraduate teaching and top 10 for most innovative universities were placed because we're young we're trying to do things a little bit differently and so that's been good for me because technology is always a part and parcel of doing things a little bit differently.

[00:04:22]
Our president is free member he's pretty you know we're unique in that he has been president of U.B.C. since 1992. I had he has been my boss reported to him since 2002 so you can see we have a stability of leadership that has happened to you and B. C. over the last 2527 years in order to be able to move forward and it's just it's always Lissa's one of the.

[00:04:51]
Places to Work. In U.S. News and World Report or The Chronicle of Higher Education almost every category we show up as a great place to work so about myself so I'm one of those that comes to a university and never leaves so I went to UN B C A 976 1st in my family to go to university ended up I was going to transfer to College Park and be an engineer.

[00:05:19]
I couldn't leave you N.B.C. because we couldn't afford to send me to College Park I would have to live on campus and so I ended up doing math and applied math and while I was doing the applied math degree it turned out computer science was in the mathematics department at that time and ended up getting applied math and computer science track degree got hired as a student graduated in 1980 and.

[00:05:48]
They offered me a job well many of you don't realize 980 had some of the worst unemployment around took the job thought I'd spend 2 or 3 years later there 38 years later on B.C. Luckily I had moved a little bit throughout there but it's one of those places where we do see people spending their life I've been the V.P. of I.T. and C.E.O. since 1970 B.C. prior to that I worked there in various roles most of my roles were pretty technical I came up through the systems programming group network engineering Unix systems administration my 1st opportunity to teach was assembly language programming then in the ninety's I did a lot of work teaching a Unix systems administration class and in our I.I.S. department I've had the opportunity to be on a number of boards from Internet to the edge of cost I.M.S. did a lot of time in the cyber security space identity management and the last 5 or 6 years I've really sort of merged into the teaching and learning space so that's sort of I've got a broad background on a lot of different things so what I'm going to discuss I'm going to spend a couple minutes just sort of talking about historical basis for technology going to go into what this next generation digital learning environment isn't isn't you know we talk about it but what is it supposed to be talk about how it's coming together and where I.M.S. fits into that and then I'm going to sort of be talking using you N.B.C. as an example in talking about how we're in visioning the next generation digital learning environment from a U.N. B C perspective and then at the end I'm going to sort of step back and from attending the sale conference I want to sort of talk a little bit with you and this will probably feed into the question and answer part about how I see some of this relating to Georgia Tech and what you're trying.

[00:07:48]
To do with sale on here and you know if you have questions while I'm talking Feel free to interrupt it's pretty informal Also I'm glad to sort of deal with it in whatever way works best yes. That sounds great guy you're probably more expert than I. Know. Back.

[00:08:17]
Up. There. Thank you. Yes No I think your vision of holistically wrapping around how you're going to support the student is really really the key in through this so I'm going to go way back you know and. So I didn't grow up with computers again you know I started college in 1976 in a personal computers you know unless you built your own computer or you got you know a little T.R.S. 80 you weren't going to be using computers before that my 1st use of a computer is the Apple 2 e and up using We had these in our micro computer lab at that point what's interesting to me is you know starting in 1977.

[00:09:30]
You can begin to see that you know technology and how it's going to be used in instruction was constantly changing my 1st job was really in what was called academic computing My job was to work with faculty in how to use computing for research or how to use computing for their courses and so more often than not faculty in the in the late seventy's early eighty's were interested in using it in their research but they were starting to bleed into I want to present this or I want to talk about this in my class but we really didn't begin to see things taking off until an S.F. net and so you know again UN B C was connected to College Park in the early eighty's as part of starting to do the ARPANET and other sorts of things N.S.F. net got set up at UN B C Probably in 1985 is when we started working on the Internet one of the areas that I spent a lot of time on back in that period was I was responsible for helping to sort of think through what our strategy was going to be for doing internet information Now does anybody in here remember go for.

[00:10:46]
Yeah a couple feet OK so go for in ways were some of the early protocols that started to come out I was lucky in that I had a next machine that I had bought for myself in the late eighty's and it turned out the 1st sort of web browsers were available on the next machine and so we started playing with this in 89 and 90 and sort of that has sort of changed my direction in thinking about how you N.B.C. was going to go and so I got very interested in the web in that early day and starting to launch that the 1st experience that I had this person standing is Murray Goldberg does anyone know who Murray Goldberg is.

[00:11:28]
So so he was a computer science faculty member at University of British Columbia and he create it web Siti and this we came out in 1996 and it was sort of the 1st way of setting up a course you know if you wanted to do a very simple L.M.'s it was written all in Perl I mean for any of you who are computer science students it was you know kind of interesting to see how you know that was was run but we load it web Siti we started running various classes getting faculty to be using this in the mid ninety's and you know you sort of solve the challenges of trying to teach you know with the Internet in the mid ninety's as you were going through that you know 2006 you know how many of you stood in line with me to get the 1st Apple one you know a phone that came out you know there and then we continue to sort of go on so from 2006 that you can see it down at the bottom but that's a screen that shows data that sort of talks about the fact that in the late 20020078 we began setting up our data warehouse that UN B C and started to provide a lot more learning analytics we were mining our blackboard system to pull data out of blackboard and then you move forward to move and now the N G D L E That's sort of there and so.

[00:12:49]
As we're moving through this I think what we're really thinking about is how we can make things richer and so the key takeaways I sort of note it is is well you know we don't worry about devices you just assume everything is Internet connected and it's going to be web excessive bore going to be mobile accessible for this we've really seen the importance of content contents always been the key differentiator in the in the Internet age and I think that's part of what you're seeing in your program that Georgia Tech is doing this program it's a different type of content people believe them what might be if it's western governors or someone else and so to me you're helping to sort of blaze a trail where you're having research faculty thinking about this content and what that means from an educational standpoint.

[00:13:42]
I really believe data analytics is going to be one of the key things that we have to be thinking about over the next few years I'm going to talk about this more as we go through this and then this idea of personalization and user center design I would say we're still in the era where a lot of these things are clues they're not see when you go from the L.M.'s to certain other kinds of tools you don't get the same preferences you don't get the same user experience this is an area where we really need to be thinking as we go forward how we build standards so that my set of preferences will move with me to whatever I want to be going with and I think that's an area where there is ripe for really moving forward in standards over the next few years and I'll talk a little bit about that towards the end.

[00:14:42]
So how did I get involved in this so turns out I was chairing the in common Federation in common is our US higher ed Federation for single sign on Georgia Tech is a member 700 is about 700 schools now are part of the Federation it's been very successful we began talking with Rob about what I.M.S. was doing we were trying to think how we could integrate federation with L.T.E. I at that point in time and we started having discussions and I really began to find Rob to be very visionary in what he was thinking about and so I knew Malcolm very well and we started having conversations about where learning was going and what were the challenges and you got to recognize this is 2012 where we're having these conversations and what we were saying was is that learning really.

[00:15:42]
It was siloed you know if you think back 67 years the L.M.'s vendors wanted you to buy an L M S product and then you were going to live with the tools and the vendor relationships that that Ellen mess vendor had established it was what we began to call a walled garden that you would always live in one ecosystem and if you didn't like that you wanted to use a tool that was in a different ecosystem you were out of luck and that really wasn't going to be what was best for education and so we wrote this article on a new architecture for learning and we sort of started with the idea that.

[00:16:27]
Right now in 2012 or 2013 learning environments were too brittle and there may there's still too brittle today often if a faculty member wants to install or bring new content into a course you might be asking that faculty member to bring you that content months before the class starts because all we've got to integrate it we've got to test it we've got to validate it's going to work we can't just let you decide a week before you're going to use it in a class that we can integrate easily into our content.

[00:17:05]
And so what we were saying is is that you need to have a way that you can easily integrate applications in a day rather than months it shouldn't be this hard we need to be able to say that faculty can take control of being able to do this that you should need a system admin who runs an L.N. mess to have to work with the vendor on the other end to be configuring and setting up a product on your behalf and that the analytics that came from that 3rd party product had to be able to flow back into an environment so that you could create a 360 degree view now those 3 statements they seem like you know common sense today they were you know.

[00:17:55]
Heretical. You know back in 2012 because we just weren't doing it we hadn't thought about what this would mean or how to even set up to do this and this is where the connection with I.M.S. comes in because to do these things it takes standards it takes a common approach that vendor suppliers L.M.'s vendors can all agree on if you're going to begin doing these kinds of things at scale and so that was where.

[00:18:26]
I.M.S. really was of interest to me is thinking about that particular way of leveraging the fact that we could be doing using standards to scale this and so that old architecture everything that you did was completely separate you know if you had a classroom capture system it was completely separate from your L M S system it might have integrated and it might not have integrated if you were using electronic text it was doubtful that they integrated into your learning environment certainly not seamlessly and clicker in assessment systems everything was completely a standalone system with no way of sharing data across that and what the article goes on to sort of talk about is this environment where you're connecting learning environments and tools and content back with enterprise systems and other systems and again in a way this is where you're going now with your sale vision of thinking about business services student services on how this all wraps in to be supporting students to be making sure that there is successful as you can be and we laid out this sort of vision for how this could be coming together and through the middle we really saw that what had to be the focus is on developing standards that we're going to enable the.

[00:19:59]
Activities to happen and we'll go in a little bit later we'll talk about some of the standards that have come out and how they're evolving to make this happen but this idea of being able to have a 3 $160.00 degree view of a student but to be able to think about this from everywhere is I think the unique piece that we tried to lay out at that time.

[00:20:23]
So fast forward a week a year and a half later. Rob Malcolm with other people for magic cause they get a Gates grant and they come out with the next generation digital learning environment what's interesting is is that as you see the report that came out on this I think it was yeah April of 2015 so it's about 18 months later and did anybody participate did you a participate in any of the focus groups Pam that they held for for the next gen digital learning environment.

[00:20:59]
You know because I know edge of cause ran a series of focus groups because I participated in a couple of them that were there and I just did know but they ended up bringing members of the community together for various what if scenarios what what doesn't work today what would you like to be seen happen what do you need to be able to support faculty in the classroom and to be able to innovate and so that ended up being sort of the basis for where this next generation digital learning now the definitions that.

[00:21:32]
Malcolm came up with I think are kind of abstract but they again they sort of fit to where you're going thinking about sale it's really about they use a confederation of I.T. systems you could call that an ecosystem and it's going to include content it's going to include analytics and it's going to have applications and digital services that are all going to come together to work seamlessly on behalf of the user that was the 1st definition they also wanted to highlight that this next generation digital learning environment isn't the L.M.'s 2.0 You know the D.L. amass is incredibly important but the L.M.'s shouldn't be trying to do everything that we want to do in a learning environment there are functions that the L A mass should be sort of tailoring down on and really making sure it can do the teaching and learning aspects very well some of the other aspects where they were starting to go into we believed that they probably should be looked at in other areas with other tools at that time and maybe the L M S isn't there to be trying to do everything which was the direction that many of the vendors were trying to go at that time.

[00:22:59]
So the N.G. Deeley ends up having these 5 areas that they highlight interoperability and integration you've got to be able to easily and seamlessly bring tools together to be able to share them and integrate them very easily. Personalization you know we need to be developing ways that and personalization takes 2 dimensions it takes the user experience dimension but it also takes the fact that learning is a personal journey and as you think about adaptive learning we need to be thinking about creating learning content that allows students to be taking advantage of the fact that we've got a wealth of content and as they hit a concept that maybe they should have mastered but they didn't they can easily go back and get more information and maybe build themselves back up to now understand that concept deeper and so adaptive learning systems really were going to be a key element and then we talk about analytics advising and Learning Assessment these 3 elements sort of come together to be thinking about pedagogy what's working how you should be moving forward to advance that collaboration was highlighted in the fact that increasingly students have true that their use to using and we shouldn't be for closing their ability to bring what they already know and like into the learning environment and so we should be thinking about being open ended and then lastly but certainly not least excess ability has got to be right from the beginning and thinking about this we need to be having a broader vision for how accessibility is going to work and what that means in a learning environment if we're going to move forward and when we begin to think about accessibility you know some of the examples that I've seen that that would be fabulous So let's say my native language is Spanish or Mandarin.

[00:25:22]
I may want to be able to get video content delivered to me in that language versus English now we think well that's very hard to have all of these different things why I would sort of go back to Georgia Tech and say I'm going to bet you in the next 5 years you know we'll have automatic language translation being relatively good we already know Microsoft can do it on spoken word where they they do the underneath of it in different languages will have this capability of being able to provide you know audio in to what you want to hear in the language of your choice that is the kind of thing that I ought to be able to set a preference for and now I can consume content if it's been set up automatically to give me in the language of my preference and so these are the way we were trying to think moving forward.

[00:26:18]
So I mean 2nd. So I think you get a sense of why I think it's important. So for for me the what has really been. The interesting question and. So. Our president has a vision that we can do better in this country in STEM. And that vision comes from the fact that there are very few places where stem actually is done well and his his argument for this is if you look at data.

[00:27:12]
There is no he studied stem through work that he did under Obama's There was a Department of Ed group that was looking at STEM education in the U.S. and what they saw is that when you look at students who start in STEM and finish and stem there was no group of students.

[00:27:36]
That ended up achieving even a 50 percent success rate in STEM. You would say that. Asian students were probably the highest they were about 48 percent white students were in the upper thirty's. African-American Hispanic upper twenty's low twenty's in those ranges we were doing an awful job even at universities that graduate 95 percent of their students they don't they fail them in STEM and the classic example that we see is I'll pick Cal-Tech or MIT.

[00:28:23]
Most of the students who go to those have almost perfect math S.A.T.'s yet they grade on a curve in their. Entry level math classes these students have never gotten less than an egg. You know and so the 1st time you're in an entry level stem class and you get a C. or a B. You're changing majors you know and when you when he would talk to those universities because we were sending our students who got bachelors to Cal Tech to MIT they're thriving in their labs and they're asking how are we doing this and the conversation that was happening was is that when these students would go to those schools they were getting a B. in that early class and that was sort of OK you should stay you should go into financial economics you know we see so many students wanting to go into financial economics to work in Wall Street or be a client that we weren't seeing they idea of how you should keep them in STEM and so one of the things that we really want to think about you know B.C. is how we can do stem better what's that going to mean at the calculus level you know when we look at our STEM for Calculus our upper core tile looks a lot like your students in the upper core tile but our mid levels are a lot lower because you don't go down as far as we do in terms of your S.A.T. bands and so the question that we've been asking is can the support been student that got 606251 their math S.A.T. and was a solid B. student in their suburban school district can't they be successful in STEM and a lot of places they can't and we think we should be able to redesign our teaching environments to get them to be able to be successful if they are motivated to be successful and not a lot of it comes down to motivation and how hard you're willing to work but one of the things that we're trying to do is we want to understand where students are falling out and that gets to number 2 we have to have a robust data environment we you know we need to understand what students are doing in the classroom.

[00:30:38]
More than we do now. Doing a pedagogical innovation and then finding that. Students really aren't consuming that innovation are picked flipped classroom. We think flipped classroom should be a better way of teaching bring students in they prepare the night before they're in small groups they're working on problems there's a lot of back and forth that should work better sometimes it does sometimes it does it we don't really understand why it does for some students why it doesn't for others data can help us begin to understand these things and so we need to be thinking about how we create data environments learning is personal you know what we see happen in a lot of instances when failure in STEM is that students had a bad teacher in Algebra 2 If a student didn't really master Algebra 2 when they get into calculus when they get into some of their physics classes that's where it begins to be a weakness that would be there we need to go back and try to see can we help them master what they didn't learn and there's some software that we're you know looking at like Alex and others to sort of see can this be a possibility and then lastly we want to unsent faculty to innovate in the classroom and so by giving them more data we think we're giving them the opportunity to say go at it this is your opportunity to to think out of the box and try to do things a little bit differently and so that's why the N.G. deal is important to me it's it's something that allows us to bring these elements together to support innovation in the classroom so a quick example.

[00:32:23]
And we have a lot of things that are out in your N.B.C. off of an analytics website called do it B.C. diety to use slash analytics. But one of the things that we're doing in this sounds stupid when you think of it in faith on face value but what we wanted to do is we have vital sources our electronic textbook provider we have blackboard as our L M S provider we wanted to integrate Analytics from the e textbook with blackboards L.M.'s how much they're using the alum asked how they're doing on quizzes with also student grades coming from R.S.I. Yes We just wanted to know you know what we're really trying to understand at the end of the day is a simple question Does reading the textbook matter now if you're a faculty member most of you will raise your hand and say yes but you know what a lot of students we find is that a lot of students don't get into the textbook and so one of the things that we ended up seeing as we ended up doing this data integration and we went forward so we did a set of classes these classes included our calculus for non STEM majors math $155.00 physics $122.00 is our standard electricity and magnetism class.

[00:33:48]
Stat $355.00 is statistics for biology chemistry majors that would be there it also is a preliminary from entering the master to sticks area and then we had foreign language classes are Spanish to a one in a psych class as well so that we're in this and what we looked at is what were the regular pass rates and things like that that came out what we ended up sort of seeing as we went through this is.

[00:34:19]
That. Not surprisingly again this is just stating the obvious but it's nice ultimately to have data to prove the obvious which was students who came into the class with basically a C. grade point average. If they didn't open the textbook in the 1st 4 weeks of class they had about a 37 percent chance of passing the class go over to.

[00:34:48]
This Yeah basically this band of students if they didn't open the textbook in the 1st 4 weeks of class we sort of knew it the 4 week mark they're gone you know in through this what's going to happen is they get hit in that 1st exam which usually comes between the 4th and 5th week mark they hadn't been prepared for what they're going to be getting on the 1st exam and usually they do poorly and they don't end up staying in the class if they had been using the textbook they had 90 percent now one of the things that we're starting to do is we're starting to capture little data systems like this week another area where we've done a lot of research was looking at students who are taking math a 2nd time what we find if a student is taking math a 2nd the same class the 2nd time we only give you 2 times to take a class if you don't pass it on the 2nd time you have to go through an appeal process which you may or may not be granted.

[00:35:50]
What we found is that if students started working with our learning resource center from day one the 2nd time they were teaching class they had a very high probability of passing if they waited until after the 1st exam or they didn't participate with the learning resource center they had a very low probability of passing and so it was one of those examples where when we 1st looked at learning resource center data we couldn't see any difference it was making on students because frankly the the students that use our learning resource center are mostly our pre-med pre-professional students who are trying to go from a be doing a you know it's all about getting straight A's it's not a.

[00:36:35]
Passing the course we don't have to worry about them what we weren't getting is the students who really needed to be using it and so we started a nudging campaign once we collect data we start nudging them through alerts and other things hey did you know that if you do this by the way if you don't pass it this time you have to go through an appeal process just letting students know where we're creating an environment to try to be supporting them was helpful in this and it was one of those sort of insights that we wouldn't have gotten if we couldn't have looked at the vital source states in the past we just didn't know who was looking at the textbook and who was it everyone assumed if you assigned the reading material it was being read.

[00:37:21]
So that one. So the other one then I'll just say before I get into this when. We're doing a another project this year in this semester with about 12 courses where we're taking we're working with a C.T. and this one is an interesting one because a city has a survey that will look at students and what it has been validated on in the case to 12 space we haven't validated it yet in the higher ed space but it tries to look at is grit and perseverance sort of this internal I'm going to push my way through problems issue and what we're trying to see is based upon how students score in that how do they use the electronic textbooks and the L A mess how do they deal with issues when they don't do well on a problem or they don't do well in a quiz in the L A mass does that double their intensity or do they drop off what we're trying to understand is is can the 1st test from a C.T. tell us something about how students are using the L.M.'s and do those behaviors in the L.M.'s and electronic textbook align with what we would expect for those attributes that came out of the A C T So it's sort of across validation but again this kind of activity you could never have done unless you're beginning to instrument the data environments that are there and begin to let you know try this out in different ways.

[00:39:00]
Sure. You. Know so so in with the electronic textbooks every time you open a page every time you highlight something we're seeing what you're doing and that electronic textbook and so it's real data that's coming in and what we we noticed is you know just actually starting to get in and read the textbook was really the key thing into that one and.

[00:39:43]
Him. Or me or. So I because I'm guessing that it's a bit of both bit because this is used in the K. through 12 quite a bit as part of a survey that a.c.t has been doing for about 5 or 6 years it's not been used in higher ed and so what we were going to be doing is we're just sort of testing out Arizona State is another one that's trying this out with them and they're going to also be presenting some results back from this.

[00:40:22]
So how should universities be preparing I think I'm having a reasonable timing in this market so I just serve OK. So I think I.M.S. really has a key role in helping you think about the next generation digital learning environment so so that I am US really focuses its technical standards on connecting to either software digital resources or tools so we have a whole set of standards that I'll talk a little bit about that work with that L.T.E. i is the one that most people know because that's all motion because it is now in the tool space we've just come out with a new update to the standard called L T I advantage which we think is going to be a game changer because it allows you to do everything without having to do A P I calls.

[00:41:21]
You can now seamlessly integrate things then you can transfer grades you can have identity L T I advantage is something that just was released literally in the last 2 months as the final standard the good news about this is all 3 of the major Ellen mess vendors were at the table working with us on the implementation so we know all of the L.M.'s vendors are committed to supporting this and the tool vendors will be able to now take advantage of it the 2nd part that's there is how to enable scalable data informed instruction and learning that student center and personalized.

[00:42:06]
And that's a lot harder because now you're getting less into just technical activities but also into how you supported each NG and other ways of doing things but what we're thinking about is we can be helpful in helping universities instrument their environments to be able to bring data to bear and we can also be helpful in thinking about how standards like L T I advantage can be extended to support this personalization profile that could flow with users so that they can get content as they want to and so I know most of you probably can't read this given I didn't realize the project or thing that was here but this going around the side are the 5 dimensions of the interrupter ability personalization analytics accessibility collaboration and accessibility and universal design and then all of these items are functions that sort of align partially to different areas within the N.G. D.L. E. And so what we're really thinking about is how we can support everything that's inside that circle better over time and what that means is that we have sort of built out a set of key initiatives to try to be doing that and so the 1st one that's up at the very top is thinking about digital curriculum.

[00:43:42]
So the standard you know it's being commonly use I am a standard. 3 as came out a few years ago how you do question answers Dick U.T.I. standard which allows question databases to move from one L.M.'s to another L.M.'s is an I.M.S. standard that's there that falls into the assessment element when you started having lots of testing taking place we came to the table to help with how do you build out standards to allow these things to move not just from L M S to L A mess but also internationally across languages and things like that digital credentialing I.M.S. own the is leading the open badges standard so we have the open badges V 2 standard which just came out in the last year micro credentialing is one of the elements you know Pam and I were talking a little bit about how it can provide insight into the skills and outcomes that you're expecting to come from courses and be able to track that across the student activities where you have pieces using digital credentialing is universities are about an academic experience but they're also about a CO curricular experience being at a university you're part of a community you have interned ships you have clubs you do leadership activities we've never really tracked those items and this digital credentialing allows us to look across the holistically at the student experience that would be their learning data and analytics I'll talk a little bit more on that learning platforms apps and tools that's L T I L M S's all this integration capability that's sitting there all of these are the standards that Ela that I.M.S. is moving forward.

[00:45:42]
And my reason for wanting to reach out and come down here is we need to use cases that push the boundaries if you're going to see standards evolve over time you need good use cases which you might not have thought about I think Salle is one of those places that you're going to develop some good use cases that not necessarily everyone is thinking about Arizona State's another school that's pushing the use cases and there have been actively engaged when you try to teach $100000.00 students online that also generates a set a use cases that would be there southern New Hampshire University is generating some interesting use cases as we start to look at it we need people who want to push the boundaries to becoming diameter so we can be building out the standards so that all suppliers can be supporting them effectively.

[00:46:36]
Not going to go too much in detail I think I've talked a lot about this I'm going to spend a little time sort of talking about how we're planning you N.B.C. to be doing the next gen digital learning environment so the 1st thing which I think you sort of gather is were instrumenting a learning environment and I'll show you a little bit how we're doing that in a 2nd the 2nd piece is really building out a very robust learning record store in Amazon and I'll show you an example of what we're doing it's really been amazing what we've been able to achieve at the cost points that are there we've been spending time working with student affairs and some of our academic programs and thinking about how we want to launch micro credentials on campus and that's really starting to come to fruition right now where we're starting to deploy them a lot faster we're looking at working with our student affairs office one standard that's called the Comprehensive learner record and you can think of a comprehensive learner record as something that integrates your academic transcript with all your micro credentials now where you're all trying to go through a block chain on top of that so that they can take the comprehensive learning record and share selectively with selected parties that's how you're going to scale these kinds of things is thinking about how we build technologies on top of standards to be able to now do new things we have a large and I each grant for sort of say developing STEM programs especially for transfer students to move them into graduate school and so we're building out because we get transfers from all different community colleges we find that there are different places and so we're building out some adaptive release software that we can be presenting to the.

[00:48:34]
So that wherever they come to us from they have the capability of getting to the same level of mastery through this and then we're working with black board to launch their ally product to improve their R. accessibility of materials and then lastly we're starting a pilot project with faculty where we can start to say we have what's called a rebel ski Innovation Award fund that allows faculty to get some money for doing innovative activities in the classroom and more launching a series that will be around instrumenting activities and allowing to give them learning analytics and customer reports and things like that when what's happening from a data intensive standpoint So talk a little bit about our learning record store so we launched it in the early fall last year out in a WS for the fall we ended up doing about $140000000.00 of that it's that came through most of that was in the blackboard caliper events that were there we wreck says our data warehouse we're importing now our data warehouse into the learning record store so that we can add context to the learning data that we get.

[00:49:51]
My you in B.C. is our student portal everything goes through our student portal and so we're interested in seeing how students use our student portal because that's how you sign up for events and other sorts of things so we're loading that data in. And what's interesting and I'll show you the 8 of us model all of this is costing me about $150.00 a month out in need of us to be doing this and and I have a great enterprise architect and what he did is he basically leverage the fact that 8 of us can do all of these things with the building blocks that E.W.S. provides And so what we do over.

[00:50:34]
Here is you know Amazon Cloud front just sits there as a caching engine blackboard vital source other tool providers are going to stream us a bunch of a vents periodically we wanted to make sure that if we were somehow down we didn't lose the stream that they sent to us and have to go back and say hey could you re send this because they're not designed to do that so sticking Amazon Cloud front in front of it allows us to cache all of these things automatically the stream is really going to an A.P.I. front end which is just an H.T.T.P. POST And so the A.P.I. is just really pulling it out of cloud front is what's happening at that point.

[00:51:17]
8 of us lambda is just a way of having some basic service is anybody use a couple of you you know we have some simple Python code what we get is we get a batch of thousands of events at at a time what we want to do is unpack them so we just run a small Python script on a slammed which takes the you know the 00s of events and breaks up into discrete events we send those discrete events over to Amazon can Isa's firehose and we load them out into S 3 we do that because we may want to go back and look at the raw data at some particular point it's cheap to store things out in S 3 having it there if we ever decide we want to just the way we do things we can just reprocess our data.

[00:52:07]
8 of us glue is just a tool set that 8 of us provides that literally is providing some other tools to process the data and so what we do with the glue is we go through and we look at what the structure of that data is and from 8 of us glue it loads it out into redshift which is just a basic database that's out there glue though is determine.

[00:52:34]
Saying did we get a different structure coming and if so it will add that field into the tables that it's loading out into red shift we don't even have to be worrying about having a pre-defined data model for what we're going to be sticking out in red shift just as vendors change it was interesting blackboard went from sending us caliper one Dato to one dot one events and nothing broke even though the structure changed somewhat the glue just said OK these new fields have been at it what demin to the table structure boom that's automatically happening now what we're going to be doing is we're putting a visual front end one to what we want to be using redshift for so we'll probably add tools like Tablo but we may also just stick Jupiter notebooks or our other things out there to be processing the data we don't know where we want to go we don't know what faculty oneness to be providing them we want to be able to have the flexibility whatever 8 of us or some vendor is providing we want to be able to stand up those tools on behalf of the faculty and let them dig in and see what's there but what's interesting about this is one person was able to set this up one person's was able to maintain it it's not a full time job you know we're now looking at this environment and we're saying this could be the environment where we load all of our traditional data warehouse analytics out into here as well and now we might be able to give people more flexibility in looking at the toolset or how they want to process this and so it's really causing us to step back and rethink the way that we're looking at how we want to do analytics on the campus going forward.

[00:54:28]
This is just a basic slide that sort of shows the vision the the whole idea of caliper is it's just a way for things to get data posted you know streaming data and so you know there could be all sorts of ways that you do this you might decide that you've got a tool that's a software development environment and you want to be capturing how much the students are doing in that software development environment All right you could send it as caliper events and then you could be consuming it if the instructor was interested in how are people using the software development environment that we've given them we could be seeing them what that means.

[00:55:13]
So that's really sort of the caliper side before I jump to the last 2 things are there any questions on caliper or what we're doing with up. There's. So I would say yes and it's a challenge so so in all honesty we are. I think we are the 1st customer to consume blackboards caliper data.

[00:55:46]
And we will find inconsistency is and we let blackboard know and they're fixing them you know in through this and so when we did vital source what we found in vital source was vital source was sending us information as we started to look at it we realize they track everything with their own internal id Well that's not very useful for you having your own internal ID How do I map it to the user that I want to compare to what they were doing in the L A mess so I had to work with all we have a flat file so we end up having to have a separate way where we pull a file from their analytics platform in that has the mapping between this and so it's been a little bit of an interesting journey but that's again part of why you know schools like Georgia Tech you can push the Omble open through this get vendors to help with this the bug these sorts of things if we're going to make these environments work we've got to be having people really dig in and use them and push the envelope so to speak but you're right it has been a challenge on this the other thing that I wanted to say as well while caliper is the I.M.S. standard one of the projects that we have going on in I.M.S. is there's another standard called X. A.P.I. and the reality is is we don't care you know we want data to come in we want to have as complete a data environment as we can if data has to be loaded through X. A.P.I. will do that we're looking at ways we think there was a working group that believes that we could probably add functionality over time to caliper to be able to support most of the common X. A.P.I. recipes and we would like to see that sort of happen and that would be on our part to be making it so that you don't have to be changing what you're doing in X. A.P.I..

[00:57:45]
The last 2 things I'll do and then I'll get a little bit to sail and that will probably lead to the questions part is. My credential really just sort of launching the micro credentials activities we're starting projects with we've been using credibly and portfolio as areas where we've been storing these in we also though sort of consume and load him into our data warehouse so that we can be looking at how these credentials sort of impact what is going on within this I think over time this is going to be one of the areas where for things like accreditations where a bet wants to know you've gotten the sets of skills we can be noting the skills through various classes and activities and produce something that would be like a comprehensive learning record to say yes all of our engineers meet the standards because we can show you how we map these skills into the area that would be there and that's sort of the vision that we have for micro credentials I talked a little bit about the comprehensive learning record I think that this is really one of those areas that we need campuses to be thinking especially ones that are trying to push the CO curricular side of things a little bit differently.

[00:59:01]
The one thing I think that would be interesting here for Georgia Tech in the cell space is just. Lifelong Learning to me you know allowing employers to be able to push credentials will be a you know micro credentials and then maybe Georgia Tech is is updating that and showing it was verified by the employer for training that they've done this could be a really interesting thing for you connecting to your alarms over time as you think about this lifelong learning space that's one of the areas we're trying to think about it on so how might this be for sale so I'm going to switch real quickly to to something here but I'll take questions while we're waiting.

[00:59:46]
So that any questions that have come through on the chat or anything else before I just start jumping in for the. No of. Pardon. Yeah. OK. Personalization and. There was a project that it was funny I went out and I could not find it on the the I.M.'s website it had been on the I.M.'s website but it could have just been they've changed things around but Georgia Tech had a 1st in the world Grant maybe 4 years ago where it was looking at accessibility and it was developing.

[01:00:30]
See what was a profile for all of the kinds of things you needed to be thinking about for accessibility one of the things that's really interesting to me is now with L T I advantage we have the opportunity to do by directional the the L.M.'s can be sending information to the tools that would be here's your profile and to me this is one of those areas where thinking about user interface and at least accommodation inaccessibility is an area that's right I'll throw out you know I don't doubt if you have an i Phone do any of you use the excess ability feature in your i Phone I might be the only one so what a lot of times that all my classes I like to go to read my text messages I make my font a little bit bigger that's under the excess ability preferences it's great when apps automatically use the features that are there because now I don't have to worry about the fact that I've got to go in and make it big or this or that we have that kind of capability before us if we could think through what does that mean for person for accessibility and I think some of the work that Georgia Tech did in that 1st in the world Grant should be revisited at some point in time so that's one area that is is something I know you talked in Sale about advising degree navigation what's really the key is.

[01:01:57]
Compre the comprehensive learning record is one of those things where. It's not just about the classes you take and getting through those classes that's important because getting a degree is critical but if your career goals are to be doing A B. or C. there's all these other things you need to be doing while you're in college to get to that point that you're going to be able to do that so so the example I like to use is we usually send a couple of students on to Google every summer I'm sure you sent a boatload you know into this one of the things we found is if students don't take data it don't take our algorithms class in their junior year they don't do well on the interview process for Google because Google is interview is really about algorithms and if you wait and take it in your senior year you're not going to be doing as well in the interview process so for students who want to do you know they're thinking that I want to try to do the Google interview we're trying to be thinking how can we recommend you've got to be taking this in your your Spring or Fall of your junior year because this is going to be important for you if you go for the interview process that they do so if the little things like that that we think we can be helping people on as we move forward the lifelong learning I think again I mentioned this is part of the comprehensive learning record I love what you're doing with block change I have a link into this doc that talks about the fact that southern New Hampshire University the MIT startup learning machines are doing a block chain effort and I.M.S. has been participating in that how block chain works with the comprehensive learning record may or may not need to be a standard but if you're all using the comprehensive learning record it would allow multiple types of block chain tools to be consuming the same thing because the.

[01:04:03]
Pretense of learning record is made up of discrete components that really look like open bags V 2 components if you want to think of it that way and so the I.M.S. standards are really key to whatever you want to use in a block chain world that would be there.

[01:04:20]
I talked a lot about that application integration earlier but this L T I advantage is really one of the game changers that we think is going to be there and if you're going to write any tools or do any development at Georgia Tech making sure you're looking at the L.T.E. i want out 3 standard would be one that would really sort of set you apart and put you in a different spot identity standards that's less here I think one of the areas that we were talking this morning that's right for research is how you look at adaptive learning questions if you think about it in adaptive learning is really an expert system that's wrapped around question answers and so what we don't have is good ways of extracting some of that question answer models from one tool to another tool and so you almost have to start from scratch if you want to go from using Acrobat T. to some other tool that would be there there's an opportunity for thinking about a standard at some point in time we're not exactly sure what that is but that might be a spot where Georgia Tech if you're doing anything with adaptive learning could be helping with sort of thinking about this because I think a lot of it is going to be how do you expose the underlying expert system into this as well.

[01:05:46]
Speech processing is one where when we talked about in the sail conference we know that the next generation user interface is going to be voice it's starting to happen in dribs and drabs now but how we make this integrated into the learning environments we haven't fully worked out yet is this going to be an L.T.E. i interface it's doubtful now maybe Georgia Tech would do this where you'll stand up your own N.L.P. cluster and be processing the voice and trying to to do that but most are going to consume services from Google Microsoft or Amazon you know that's what we're seeing happen in the world is that vendors are going to be providing voice to text if you want to think of it and how do we then interface that into our learning environments for people to be able to use is one that I think is is really interesting the we talked a little bit about intelligent tutoring the adaptive mechanism intelligent search is another one so so we're just getting ready in I.M.S. to be launching a new L.T. I resource for searching across L.T.E. i excessive all resources this is going to open up the idea of being able to search across lots of different content repositories that you may own you know it's not the traditional Google search but how do you look across those different things that you alone for the best content I think there's how we make it relevant what the relevance algorithm is going to be is going to be a great computer science sort of question into that just making it that you could do it was part of I.M.S. but how you do it well is where we need people to really be stepping back and thinking about this Was there anything sorry.

[01:07:42]
The the last one I would just say that we're really also for. Oka story on is thinking about how we do privacy management and security and that's one where we're really trying to think about how we can come up with the ways so that. Right now every school goes and does a security and privacy review on every app and that seems like it's a lot of redundant work and there ought to be able to be some way of having some commonality where we can share a best practice approach and so if Georgia Tech did something I may be willing to accept what Georgia Tech it done related to that into this or if B.C. did something you might accept that because we're sharing the same approach in the way that we do app that Inger we've got a process down that it's an independent group that's looking at it on behalf of all of higher ed that could be there and we see that as something that I am yes is starting to stand up and we think could be a way that we do this and so that's the end.

[01:08:45]
For 40 so about an hour so any questions or comments. So it's no it's a great question and I think that's the challenge of what these records are going to ultimately look at but if we think about it as a student is going to enable this so you I don't know as part of lifelong learning will you be allowing students to maintain account lifelong or some mechanism here.

[01:09:36]
OK expect to as well so if a student owns this account and it wants to put that information in their comprehensive learning record who are we to say that's not worthy if the student is deciding that this is important to me and the employer is prepared to create a badge that has a validation mechanism OK this employer validated it for this we ought to allow that I think now is it going to clutter this yes and that may be where the block chain says you don't want to be sharing this because you're going to see a lot of things that are clutter that are pertinent to what you want to share your comprehensive learning record to this other party with and so you'd be able to filter it to filter out the things that are less pertinent and be able to show the items that are but there's still a validation mechanism there but that's the way we're trying to think about it is is in that way but I think if your point is is really well taken what we don't know now is is everybody going to have a badge for everything and it's just going to be pure babble you know that you know we can't tell what any of this means and how to make it worthwhile and so that's that's you know the question that I think will sort itself out over the next 5 years you see a lot of companies a lot of groups trying to sort of lay out frameworks for thinking about the credentialing space.

[01:11:12]
He has. You know. Other questions or. Own. I get a bounce for that right now. You know. I think no not fully to all of those components that would be their ally was is the 1st tool that we found that is very helpful for sort of looking at how you have designed your course at least for visually impaired and there are a lot of things that would meet accessibility but if you were trying to navigate the course in a screen reader you were going 8 or 10 levels deep and if you ever try to use a screen reader and you see what that experience is like it's horrible and so Ally sort of highlights where you've gone into bad practice and so we think that that's a 1st starting point when some of the other dimensions I don't I think there needs to be some basic research that's probably done as to how you will sort of handle you know the levels of consumption that may be there in thinking about that and how you're presenting that information in different ways but but to your answer the question I think that's an area that's still ripe for a lot of further research and the others.

[01:13:04]
Are. Right. There. So so the question just I guess I'll repeat it just in case it wasn't heard so the question was How involved are companies like credibly learning machines other suppliers with a comprehensive learning record. They're certainly part they are incredibly active in the open badges V 2 initiative so so this open badges standard that we just launched back in November all of the companies were really at the table around what that standard is going to mean and how do we do in our operability in that instance that really forms the components of the comprehensive learning record.

[01:13:54]
They're related the group that really has been most involved that I've seen is we're seeing acro which is the registrars group and NASA which is the student affairs group have come together they have a They've had 2 large Lumina grants and what they're looking at is trying to understand how we can show the full record of student achievement that would be there and so that is really more being driving that striving to see a larger than the corporate side of things and that's probably good because frankly it should be a registrar's in student affairs professionals and others who are trying to drive what we want to think about in the comprehensive learning records we'll just be able to consume whatever is getting produced as long as that's open badges be to anything else yes.

[01:15:06]
So the data is probably a problem. Yeah yeah but yeah it's that we could debate you know it but no that would be the interesting thing I mean there's a group called unison which has come together with about 12 or 15 schools and they're building a common data link Vinces dream for the University of California system is to build a common data link I'm having conversations in the university system of Maryland to be saying maybe we do we have a mechanism where we could easily share data we report to one board of regents you could be thinking through that there's a sharing capability we want to look at this but looking across many different universities probably hard for sharing the actual data related to how we're doing this most definitely it learning impact which is the I.M.S. conference in May We're going to be having sessions on what we're doing in terms of the learning record store we think that in the past people have been standing it up in ways which is much more complex what I expect you'll also see is what we're doing there's going to be vendors that are starting to come out because it just makes perfect sense if you're used to working in an 8 of us world this is the way to go but yes we're going to be sharing what we're doing Vince is at U.C.S.D. is going to be sharing what they're doing in Google Cloud platform and so what we want to do is is show that it's a lot easier to be doing this than people usually thought it was and so that might help spur more development of setting up the data links in this environment.

[01:16:44]
Anything else and A. And exploring different. Delivery. There or that we're going to. Live. Documents that you can actually read every day that. All. Do the research there. Should be a little bit right part. Of the story. So are you referring somewhat to like Jupiter notebooks or some of those kinds of one joined up model their.

[01:17:40]
Project OK. Research. Because of the document itself. Our papers are never. Going to research you're. Doing. So so they are grabbing stuff so the. The piece and I.M.S. just sort of it as we're thinking about it is it's made up of members so our members are higher ed and K. through 12 with suppliers.

[01:18:25]
We can propose anything we want to try to propose as a standard model and and a lot of these things take you know the comprehensive learning record started about 45 years ago I think some of these product what you're sort of hearing from me is I think what you're highlighting is one of those areas where if it's some point standards are going to be important because you need to be able to go from this open document stack to Jupiter notebook and somehow you should be able to consume it in either way and have a portability mechanism between there that might be a spot where the right standards would be and where there might be some other spot that's the spot where I am less can sort of be an area where he can come to the table there's other groups like i triple E. you know that can also do these sorts of things and so we tend to work with with whatever groups where we're trying to drive it is we see our sweet spot is in the teaching and learning space and through that that's a great question you know into there because anything else well thank you I really appreciate you letting me come and talk about this.