XDBrowser: User-Defined Cross-Device Web Page Designs

XDBrowser: User-Defined Cross-Device Web Page Designs


– …they would like
to create, and I think be back here and so what I show
you is like a first version of this browser and with this browser we conducted a study and that study is the focus of the paper. I’m not sure what happened. This cable touched my laptop
and then everything died, but I’m almost back. – [Guest 1] (mumbles) – I’m almost there. So, by the way, the other
thing I wanted to say, this is a multi-device environment, right and all of you guys are
here with multiple devices. It would’ve been so great
if some of this research would be at the stage
where we can just flexibly, I could just borrow some
of your devices to help me out in this kind of situation
in which I was just in. So let’s skip this a little bit because, I’m gonna skip two slides, but that’s fine because Peggy covered it all
and so what I wanted to say is basically, I wanted
to think about how we actually currently program interfaces, and the way we do it right
now is we have this really growing device landscape
and it is really difficult for designers to create
all kinds of interfaces for all kinds of these devices. So what happens in practice
right now is they create different versions of interfaces
and they did that very well to these devices individually,
but now in addition to the multi-device interaction things, and Peggy has shown us some
cross-device interfaces but I’m really thinking
about these interfaces that can run across multiple
devices that can span across these devices. Arbitrarily, they can
involve multiple users and definitely also multiple
devices of different sizes. So the motivation why you
want to do this is basically, cross-device interfaces can
be used for many things, especially in mobile scenarios
where it is now possible to combine multiple
smaller devices to create larger screen real estate
and it is also possible to actually, not in every
suggestion it makes sense to provide voice input into a smartwatch but smartwatch is
technically, at this stage, usually bound to a smartphone. So the smartphone could
easily become an input device, to the smartwatch, like a
keyboard for the smartwatch and then what you see in many projects, if you look into the
papers is really this idea of remote control where
you use a smartphone to actually control larger screens. So, over the years I feel
these family of tools, this XD family of tools,
where Peggy mentioned XDs to you before but I’ve actually tried out other kinds of directions
of really creating cross-device interfaces
and that’s just me. The community has, of course,
investigated a wide variety. Don’t be mad at me if I didn’t
include your paper here. Thank you for showing up
and it’s a lot of work, and I couldn’t cover it
all but more recently, I think we’ve seen this direction going towards smartwatches. So what I’m gonna show you
today is basically XD Browser. XD Browser is a browser
that itself is aware of multiple devices that
the user may be carrying. XD Browser is a distributed
user interface in itself, and what you see here is that each of these browser instances, if you will have awareness of the other
kinds of browser instances that are running on other
devices as expressed by the color coding here, and yes I know, color coding is not ideal, but
that’s what we do right now. So, what XD Browser now allows us to do, if you are a Web Designer
in the audience here, clearly the title is I wanna
show you some cross-device web page designs. So we’re very much thinking
about responsive design at this stage, making sure
that the web interface looks very well individually
on each of these devices, and maybe some of you,
I don’t see this yet, we don’t have the web
browsers on smartwatches. There’s only one at this
stage but some of you will soon have to think
about how we can actually adapt our web interface or parts of it to run on smartphones. Now the idea of XD Browser
is maybe you don’t have to think so hard about it
because what I wanna do is raise all of the things
that you’ve seen previously in cross-device tool
kits, so technical support for cross-device interfaces. I want to raise it to the
level of a web browser where end-users can do this, and so then, end-users can create such
interfaces where they flexibly redistribute components of
an interface as they see fit. So I show you some examples
of what people wanted to do with the browser when we
actually gave it to them. So this is XD Browser in action, and here, what I’m showing you is some examples. So this is Gmail distributed
over two devices. This is Maps distributed over a tablet and a Microsoft Surface tabletop and what he notices here
is the same location but in different kinds of information and one of the typical
examples (mumbles) something that would’ve maybe saved my heh just now. It’s a smartphone that I
could’ve taken out of my pocket to give this presentation
to you by just hijacking the device behind me. So this is XD Browser and
what I want to show you is quickly how users in
our study or the current state of the browser, how
users actually created a cross-device interface like this. I don’t show you all the things. The scenario here is that the
browser can flexibly switch, the user can flexibly switch
between different applications running on the smartphone and the tablet via tab switching on the
smartwatch as you see on the left, but the main application
and what I want to show you is how we could, for
example, distribute Gmail so that it makes sense. You can quickly check the
inbox and then actually reply to emails and use the larger device as a keyboard kind of thing. So what the user can do at
this stage is independent of how this web interface was programmed, no programming laws required. Just select things with your thumb and move components around. If you will, it’s like paper prototyping but we’re using interactive technologies and this web interface
is of course created as a (mumbles) kind of tree. So there’s a lot of nesting
involved but the user doesn’t have to care. They only have to select the right element to just move things around. They can enable synchronization
and once this is done, as I point out here, you
can now actually interact, I’m not sure which hand to
use, but you can now interact with this device and the
user now switches between emails that may have come
in and can use the larger device for actually reading these emails and then answering to them. This is just one of the examples. Users have created many. We’ve also looked into
a variety of interfaces. What I wanted to do
here is actually choose popular interfaces. I claim these are end-users of these kinds of interfaces and what
these interfaces resemble is something like Gmail,
YouTube, and Google Maps and then the article interface
here is something close to Wikipedia, if you will
and slides maybe something like Flickr or something
that is more image-oriented. So what we did was a study
and that’s the core focus of the paper in some
sense with nontechnical, with 15 nontechnical
end-users, and they created 144 and now this is the key
point, desirable multi-device interfaces, not those that
are just technically feasible because that’s what we’ve
seen a lot in the (mumbles). Everybody proposed a tool
kit, including myself. Then I show you whether
it’s technically feasible with this, and we are all
impressed but is this really what users want and so,
I wanted to explore this a little bit more. What I noticed in this study
is for each of these five interfaces on average, people created two different versions,
and this was because in mail itself, you do multiple tasks. There’s not just a mail task
’cause then you do multiple things in mail. One is like, checking for
mail and maybe replying to it and others, a very
intensive task is actually composing an email. So we have now, based on
these 144 cross-devices designs that were created by users. We analyzed them and we
extracted six cross-device interface patterns. Some of these patterns are pretty clear. So I chose two which I wanted
to show to you in detail that are more on the paper. Interesting was to
observe how users created these interfaces. So they used primarily three strategies. Clearly, they wanted to
optimize for screen space. What I mean by this, often,
you want to combine screens to create larger screen real estate. You also want to optimize for input. This, we used mobile phone and
tablet, as you can see here, and what this meant
for most people is that they would use their
tablet to actually provide input rather than a smartphone where they don’t have such a big
keyboard and they also wanted to minimize device switching. Clearly we all know that
there’s attention division between multiple devices
and so you want to kind of even though when you switch between tasks (mumbles) One of the results of
the study was do you want to trigger different kinds
of interface depending on the task but you still want
to minimize device switching. So these are kinds of
parameters and in this space, these users tried to create
different interfaces. So the overview and detail
pattern, for example here applied to mail is the
one that you saw earlier where I’m cutting kind of
like the inbox component and the actual reading
pane is in another device and this can make sense,
for example, if you have multiple incoming mails that you want to read and reply to or you’re replying to one email but you wanna look up another email and because there is
some cross-referencing and this can be very useful. Can I just say, so what is
important about these patterns is what you see here is that the interface is really distributed. So this house supports asynchronous task. What you see a lot with
cross-device interfaces is a lot of mirroring is happening
and so, what’s the point? We don’t need to have the
same interface replicated on all kinds of devices. It can make sense in some
sittings but not in this. So we saw more of these asynchronous views of multiple devices kinds of patterns. The other one was really
clearly assigning roles to different devices. So a popular one was to
actually assign the role of providing input to the
tablet, but as the user actually provides input on the tablet, you can write an e-mail here for example. This could also then
feedback into the smartphone, and what I’m gonna show
you is, I wanna discuss some of these things,
and then I wanna tell you what I’m gonna do next
because this study is like the first result of a larger project. So, this is at the same time an end-user customization tool. So my point here is that
we’ve seen lots of tool kits out there and whenever I
do studies and cross-device interfaces, I always get
feedback from reviewers, which is valuable, but
they always ask me to find these cross-device developers
and do studies with them and frankly, this stage
is a struggle because we don’t have this
generation of cross-device interface designers. (mumbles) you have interface
designers that care very much about interfaces adapting well to different kinds of
devices, but not the kinds of cross-device experience and studies. It’s just difficult. So my approach here was
to study with end-users. So we need to enable them,
and a tool like XD Browser can do this. Then what is important, and I
think one of the main results is even though we’re creating
our tool that’s relatively flexible and we saw lots of
different kinds of interfaces for five popular kinds of applications, it was really clear from
the user study that we need to be able to easily
trigger and switch different patterns, even while
somebody’s working within one application and becomes
even more interesting when they work across
multiple applications. Users want to be able to reconfigure these cross-device interfaces. What this tells us is even
though this tool kit approach to cross-device interfaces is valuable, designers will probably
never be able to fully anticipate all the kinds
of devices, all the kinds of device configurations and
all the kinds of interfaces that users may want to have. So a tool like XD Browser,
which is an end-user customization tool is very useful. At the same time, users
wanted my browser to do more things more automatically. So even though I tried
to make it very simple and move things around,
you know, selecting and moving, they wanted
things to be more automatic and I myself want things to
be more automatic because I don’t want it to become
an authoring kind of tool, it should still be a browsing tool, and they want more (mumbles)
control when it comes to synchronization. I discussed this. It’s an interesting aspect,
specifically for people that create cross-device
and tool kits and so this is extensively discussed in the paper. Where to go from here? So I wanted to show you two things that I’m currently interested
in and will continue while I’m at Michigan. So, where to go from here? The first thing here is
I’m creating currently XD Browser 2.0. By default, it actually
duplicates, it’s like a mirroring by default but once the
user decides to double-tap into a component. Something we’re very
familiar on mobile devices, the zoom-in component. XD Browser would automatically create this cross-device interface. It is basically an instantiation
of one of the patterns that we’ve learned from the
study and the key challenge now becomes to extract the relevant (mumbles) webpage (mumbles) elements and trigger the right pattern
and this, the user here can choose whichever
device in whichever order and depending on the element
that they double-tap, could be a video on YouTube,
people would then apply their reasonably good
cross-device pattern. The other thing is even
though my work, most of it has been happening in
browsers, the other work, the other direction that
I’m following is how does this apply more broadly? So this is gonna be a bit confusing, but here you have like a
smartwatch, smartphone setting. The user just replicated
one of the screens to the smartwatch. Now the user starts to provide
input on the smartphone but then decides to actually
provide speech input on the watch and it can also
be done the other way around, which is what I’m gonna
show next is the user starts a task on the smartwatch, wants to provide input,
but this is probably a longer subject. So they just used the phone as a keyboard, and if you found this video
confusing, this is good, because I wanted to show
you the flexibility that we can achieve by making this possible in a cross-device scenario. That’s the point I wanted to make here. So, I think, I’m not sure
how I’m doing on time but, sorry for the technical
problems in the beginning. I’m happy to take your
questions now, okay. Thank you. (audience applauds) – [Guest 2] (mumbles) Telecom
ParaTech, thank you very much for this talk. I really like this work. I did want to push back a
little bit on the assertion though that it’s desirable interfaces. It strikes me that these are
more desirable partitions of existing interfaces. For example, I was wondering
if, do you have a sense of how you would be able
to, for example, prototype or model something like
say, shaking something on your watch or doing
some sort of interaction that the underlying application doesn’t already support? – Yeah, so two interesting
things you’re pointing out here. One is that the users
didn’t actually create new parts in these interfaces, so, it was not a re-authoring of the interfaces,
introducing new elements. So this becomes very interesting
in the cross-device use. (mumbles) you could think
of it more like it’s a reshuffling of the existing
elements, which makes sense, desired for these users,
given what they have, but the community out
there has thought about how to redesign a much
better YouTube interface and this is suddenly a viable direction. What this tool gives
you is you can transform existing interfaces and reconfigure them. The other thing is the
whole interaction stuff, like coming out with new
gestures that would fit in. XD Browser at this stage
is not designed for it, but it could be a browser
and there could be extensions to it and who knows what’s
gonna happen at (mumbles) next year and so, we will see. (laughs) – [Guest 2] I look forward to seeing it. – Okay, thanks. – [Guest3] Thanks for (mumbles) talk. It’s very nice to see it again. – (Laughs) – [Guest 3] Alright,
(mumbles) from Google, so I have a question. I’m wondering if you think new interaction and interface
techniques on smart displays might make some of the cross-device
patterns not necessary. So I feel the usability
of small screen input and output being improved everyday and I feel some of the patterns you have showed from the users exercise of the distribution is (mumbles)
response to the failures in designing (mumbles). – That’s true and, so I
don’t want to be negative, but I think some of the
interactions would probably not make the patterns obsolete, but they would help us
actually trigger these patterns in more natural ways because
what I’ve I seen right now is this like selecting and
moving around and double-tapping is still very constrained. So, I don’t think this is the solution. So also in addition to
what, who was before? I just forgot who asked
me a question before, but basically, this is a
new version, and I want to explore these new directions
together with the community. I’m also gonna release
this and I don’t think this covers all the kinds of
cross-device interfaces that we may want and I’m looking forward to Michelle’s question. (laughs) – [Guest 4] Hi, (mumbles) nice work. So in the scenarios you
showed, it seemed to me, I want to say this in a (mumbles)
way, but it seems static and I don’t know how people had to pair these devices together
because in the course of a day I have my phone, okay, it’s
my phone but I have several tablets at home and at
work and so computers and maybe the TV and things
like this, so how do you see these patterns work
in a dynamic environment? – So this is a study we
decided to have it more like a lavish study, even
though, until that’s why we, for example, we didn’t
introduce new devices while we were doing this study. We just gave them these two devices. So related to your question
is like, what if we add other kinds of devices or
other types of devices? Do these patterns still
apply is a valid question. I don’t know, I would hope
so and then, I think because it is a browser and one
thing I didn’t tell you specifically is how it’s implemented. What it does is basically
it hijacks any kind of browser you have on your device. So this is not like the
new Chrome or whatever. It’s just using the browser
you have on the device. So I think it’s very flexible
and it does require some more later HTMLs user standards
and I did have a little bit of trouble with smartwatches
because there’s only one open source (mumbles) out there. Call to Google, maybe you should implement a watch kind of browser. So I’m not sure how this
whole thing is playing out but I wanna enable people
to use this browser more flexibly in different
kinds of environments. – [Guest 5] I have just
quick followup question. How’d you see people might say, (mumbles) those customizations
and how do they persist? How do I, next time I come, do I find the email configuration the same if I have the same devices? – Yeah so, one thing is and
it’s sort of kind of like a SmartBook (mumbles). So there’s research on that
when it comes to end-user programming and I think I could leverage some of these ideas. Now I think what would be
ideal if I could deploy this kind of browser and
really give it out to people and lots of people can
create different kinds of configurations. Then we could actually
build a community around it and maybe share it. What I would then share is
like for popular interfaces, I think what would happen
is popular interfaces and popular device configurations,
new users coming in with similar devices
could immediately benefit and then we could, what would
be a much more interesting study, I think, in terms of
scale to do in the future. – [Guest 5] Thank you. – [David] Hi, David Carter, MIT. So this is actually a very
natural follow-on to the previous question because you’ve
started to identify these sort of common patterns of
usage across the multiple devices and such. Have you thought about
whether it might be possible to sort of enrich the HTML markup in a way that describes the page
such that it can sort of automatically be partitioned appropriately across multiple devices? – Yeah, so this is, when
I showed the quick video of XD Browser (mumbles),
this is the current state that I have. So what I could give right
now is like a tool kit to people that allows them,
basically they could just annotate different elements
and based on these elements, I could trigger these
patterns automatically. So this is the stage that I have right now but what I’m even more
interested in is trying to be a little bit more
intelligent about it by applying the techniques
to existing web interfaces that don’t follow any of these patterns. So what I would say is like
this is the easy approach in some sense, may not be
true for all kinds of websites out there, but I’m not sure
whether this is really gonna be a vivid approach. So I think the more
challenging and interesting research direction is
actually to try to do some of these automatically
based on the knowledge that I gathered from this study, yeah. – [Guest 5] Thank you very much. – Thank you again.

Leave a Reply

Your email address will not be published. Required fields are marked *