Skip to main content

Augmented Reality

Harry Potter Coming to the Real World via the Magic of Augmented Reality

Harrry Potter is about to get the Pokemon Go treatment.

Niantic Labs, the Google spin-off that developed the  Pokemon Go augmented reality game, has announced they have a deal with Warner Brothers to produce a game called Harry Potter: Wizards Unite:

We’re incredibly excited to announce this next step in the evolution of AR mobile entertainment. With Harry Potter: Wizards Unite, players that have been dreaming of becoming real life Wizards will finally get the chance to experience J.K. Rowling’s Wizarding World. Players will learn spells, explore their real world neighborhoods and cities to discover & fight legendary beasts and team up with others to take down powerful enemies. We’re thrilled to partner with Warner Bros. Interactive Entertainment, and WB Games San Francisco’s development team to bring this magical and beloved series to life in a brand new way. Harry Potter: Wizards Unite will leverage the full stack of the Niantic Platform while also providing an opportunity to pioneer all new technology and gameplay mechanics.

We look forward to seeing you outside on your next adventure. Stay tuned to our social channels for updates as we work to bring this game to life.

David Haddad, president of Warner Bros Interactive Entertainment, said: "With Portkey Games, we are thrilled to answer the fans’ requests for more games inspired by J K Rowling’s Wizarding World. We are working with talented creators to build games that focus on player-generated stories, which will live alongside the magical universe created by J K Rowling."

There’s no word on the price, release date, or game play, but I for one am hoping they will find a way to enable wizard duels. Casting spells is fun and all, but playing against another wizard would be better.

Sony’s Projector Concept Turns Any Table Into a Touch-Enabled Display (video)

sony-projector-future-lab-sxswEarlier this month Sony officially unveiled a new research and development program called Future Lab, and they were at SxSW a couple weeks back to show off their first concept designs. In addition to a smart necklace, Sony also demoed a projector concept that turned a table or other flat horizontal surface into an interactive screen.

Sony called it an interactive tabletop, and as you can see in the demo video users could use the prototype projector system to interact with a copy of Alice’s Adventures in Wonderland, and other objects.

After the book was placed under the projector, the sensors recognized it and created virtual animations which the users could lead around the table. The system also let users interact with a teacup, and a deck of cards.

The Verge got a photo of the projector itself, and you can see it at the end of this post.

It looks nifty, but with Kinect and other motion-sensing devices now available on store shelves, Minority Report style virtual interfaces are old hat.

There’s very little here which is new; Fujitsu demoed a similar concept back in 2013, and so did Disney. Also, Sony showed off an unrelated tabletop project earlier this year. In fact, if we go further back we can find videos showing a much simpler version of the same idea in 2010, and earlier.

And here we are in 2016, where this idea is still very much an idea for the drawing board, rather than the store shelf.

Why do you suppose that is the case?


P.S. And to be clear, I’m assuming that this is still a concept more than a commercial reality because otherwise Sony wouldn’t have trotted it out as a cutting-edge concept rather than a commercial prototype.

If you know of a commercial product that can do this, please leave a comment.

TheVerge, TNW

Patent Filing Reveals Smart Glasses from Amazon – Their Next eReading Device?

amazon smart glassesBetween the Echo, Dash Button, and the Dash, Amazon is no stranger to novel gadgetry, and here’s one that could be turned into an ereading device.

Re/code has the scoop on a patent that describes Amazon’s take on augmented reality goggles like the Magic Leap or Sony Smart Glasses.

Originally filed in 2013,

The patent explains how the smart glasses might be wired or wirelessly connected to a device such as a tablet and display video or images from that device in front of the wearer’s eyes. Tapping on the tablet, it explains, transitions a surface in the display from opaque to transparent, making it possible to interact with the real world without taking the glasses off.

"On the one hand, a large screen is beneficial for watching movies, playing games and even reading email comfortably," reads the patent. It continues: "On the other hand, the larger the screen, the bigger the device, which may be less desirable for a light and portable product. Another problem consumers experience with portable devices, like tablet devices, is the lack of ability to immerse themselves in a tablet experience, such as watching a movie on an airplane."

Basically Amazon has a patent on a pair of glasses so smart that they can either block out the world around you or be transparent and let you see the display’s contents superimposed over the real world surroundings.

To be clear, the patent describes the glasses as a new form of display for video, but it also suggests that the same tech could be used as a HUD for displaying info (think Google Glass), or it might be used for augmented reality uses (like the Magic Leap or MS HoloLens).

But I wouldn’t expect to see it any time soon. Rather than release a commercial product, Amazon filed for the patent. That tells us that they’re interested in the tech and that they think it could be valuable one day – it’s just not ready yet.

And given the indefinite status of Magic Leap, the hottest player in this niche, that’s no surprise. This firm raised half a billion last fall to fund the development of cutting edge augmented reality tech very similar to the Amazon goggles mentioned above, but no one knows when it be on store shelves. According to the latest details shared earlier this week when the company announced it was shifting operations to CA,  Magic Leap isn’t saying when they will launch their first product (it is filing a lot of patents, however).

And so I wouldn’t expect to be reading your Kindle ebooks on the Amazon Embers any time soon.

Disney’s Augmented Reality Coloring Books Brings Creations to Life (video)

When it comes to developing new book tech, few compare to Disney. Its researchers have developed several new ideas for interactive augmented reality, and they’re back again this week with a new take on coloring books.

disney coloring book augmented reality

With its Color Alive line, Crayola has found a way to merge paper coloring books with apps to let kids bring their artwork to (digital) life. And now Disney is taking the idea a step further.

Crayola let kids bring a finished drawing to life, while Disney researchers have taken the idea a step further. They have developed tech that lets you see an augmented reality version of the drawing while it’s still being filled in.

Point a mobile device running the companion app at the drawing, and you can see the 3d character take on the colors and texture chosen by the artist while the drawing is still in progress.

That’s a nifty idea, but I don’t know how popular it will be. Gizmodo said it was "a brilliant way to bridge the experience of coloring with crayons or pencils (which has yet to be perfectly recreated on a touchscreen device) with the apps and games that more readily grab a child’s attention these days", which is true.

But with the way that old-fashioned un-augmented paper coloring books are making a comeback, I don’t know that anyone is going to want a mixed media experience.

Half the fun in filling in a coloring book is that you’re not holding a gadget in your hand. This would seem to defeat that purpose, and that is perhaps the greatest distinction between Craolya’s tech and Disney’s demo.

Crayola has found a way to add to the finished drawing, which is nice. Disney’s idea, on the other hand, feels like it complicates the drawing process – gilding the lily, as it were.

Would you want to use it?

Disney Research via Gizmodo

Sony’s Smart Glasses Will Make You Look Twice as Nerdy as Google Glass at Half the Cost

Google Glass may have been shoved in a closet, but that doesn’t mean Google’s competitors have similarly given up. Earlier today Sony revealed that it would be offering  its Android-compatible smart glasses to developers early next month.


"As a hands-free device, SmartEyeglass can be a promising product with many practical uses,” a Sony spokesperson told PC World. "But since we recognize the need to explore applications at this stage, we’re releasing this developer edition."

Unlike Google Glass, the Sony SmartEyeglass can only display monochrome text and images in a user’s field of view, and not off to one side. There’s no way to interact by voice, winking, or blinking; instead Sony went with a  a separate wired controller unit that houses a speaker, microphone, and an NFC module.

That sounds a little old school, but considering how completely the SmartEyeglass covers a user’s vision it might  not be a bad idea. And as you can see in this promo video, the physical interface looks well-designed.

The glasses also have Wifi, Bluetooth, a 3MP camera, a g-sensor, and an accelerometer. It’s going to be sold through the Sony Developer World website, and will be released in Japan, Europe, and the US on 10 March.

Sony plans to charge $840 for its augmented reality SmartEyeglass, and it is targeting developers and industrial applications ahead of a planned commercial release sometime in 2016.

If you don’t want to wait for the release, Sony has already released an SDK for the SmartEyeglass Developer Edition SED-E1. You can find more info on the website.

"Goodnight Lad’s" KickStarter Campaign Shows Just How Far Augmented Reality Books Have Come in Four Years

Augmented reality books may not have taken the larger book market by storm, but as we can see from the following Kickstarter campaign their time may have come.

goodnight lad

Good Night Lad is a new project by Apple developer Brad Grimm of Giveway Games which combines a traditional kid’s picture book with a companion app running on your smartphone. The book tells the tale of one boy’s night time adventures, all brought to life via the magic of augmented reality.

The paper book tells a good story on its own, but it only shows about half the fun. Point a smartphone running the companion app at the book and it will come to life:

I just read about this campaign today. It’s two days in, and the project has already passed its goal of $7,500.

While that’s great news for Brad and all the other parents who want to buy this book, this campaign caught my eye because of how little it cost to fund this project.

I first saw this tech in 2010 when S&S released bonus content for The Search for Wondla, a kid’s book. At that time a major publisher was required to fund the development of a web-based solution which worked not with a smartphone but with your webcam. That required installing a Flash app which, by the time I got a chance to try it in 2012, had stopped working because it hadn’t been updated.

Four years later, and not only is the tech easier and more convenient to use, it’s also a heck of a lot cheaper.  An Apple developer can crowd-fund an app for less than $10,000.

That’s not pocket change, but it is small enough that money isn’t a limiting factor any more. Now the technical and artistic skills of the development team are more important, and that means the only real limit on how augmented reality is used in books is the need for a clever idea and a good way to tell a story.

MagicBook Wants to Add Augmented Reality to Paper Books

A number of different tech companies have tried to bridge paper and digital by adding augmented reality features to paper books over the years, and now MagicBook is giving it a go. I bumped into this young startup in the Eureka Park, where it was showing off its ideas for an augmented reality children’s book.

magicbook 2

The tech is in the early stages of development, so there wasn’t much for me to see, but I can tell you that Magicbook is working on an app-based concept which could be made to work with any book (so long as the digital content exists).

All they had to demo was a couple pages made for the classic kid’s book, Where the Wild Things Are. In the demo I can make the trees grow in the bedroom, or see the monsters. All that is required for this trick to work is the book, the app running on a camera-equipped smartphone or tablet, and of course the digital extras.

That last will likely be a stumbling block for MagicBook; as we’ve seen from the general non-adoption of augmented reality over the years, few publishers are willing to invest in creating the content.

But the 4 person team behind MagicBook have high hopes that this won’t be an issue. They say that they’ve already received positive feedback from parents:




New Wearable from MIT’s Media Lab Wants to be the "Feelie" for Books

Thebeach2[1] Media Lab at MIT has just revealed a new concept for a physically interactive book called Sensory Fiction, and it looks like Sci-Fi’s longtime pipedream of programmable sensorial storytelling has finally finally come to pass.

The Sensory Fiction project is designed to try to physically make you feel the characters' emotions as you read the story. It consists of a hodgepodge of networked sensors and actuators, divided between a new type of connected book and a harness worn by the reader, which combine to create physical sensations that mimic what the characters are feeling.

Changes in the protagonist’s emotional or physical state triggers discrete feedback in the wearable, whether by changing the heartbeat rate, creating constriction through air pressure bags, or causing localized temperature fluctuations.

This sounds like it is still just an early concept design, and not the sophisticated feelie which Aldous Huxley described in Brave New World, and at the moment it can influence the reader by inflating airbags in the harness (to increase tension), playing sounds, vibrating to influence heart rate, and using LEDs to change the ambient light based on the setting and mood.

The initial project was developed for Alice Sheldon’s (writing as James Tiptree) The Girl Who Was Plugged in, a story in which the protagonist experiences life via a neural link to a body which she remotely controlled. You can see the idea in action below.

On a technical level this is a fascinating idea, but I think it may have been developed under a mistaken assumption. If readers wanted to experience the events they are reading about they would put the book down, get out of their comfortable chair, and go have adventures.

Then again I am usually a reactionary to the various attempts to gild the reading experience, but in this case I would point out that if a writer is good enough to convey the sensation with only words then there is no need for the harness shown above. And if the author is not good enough to convey a story with only words then what are the odds they will be able to write a worthwhile story which makes use of the Sensory Fiction harness?

In any case, I’d much rather see someone develop a story around Disney’s HideOut. That project uses a small projector to add content to a paper book:

disney hideout

Media Lab


Disney Unveils the Next Big Thing in Augmented Reality

HideOut-Device[1]If you haven’t gotten enough of augmented reality in books after reading about the Bridging Book, textbooks, or Marvel’s garnishments on their comic books, then you might be interested in the latest project from Disney Research.

A research team located at a Disney facility in Pittsburgh and at Carnegie Mellon University has developed what is probably going to be the next great innovation in gaming controllers.

They have a prototype handheld device that combines a projector with an image sensor. The HideOut, as they are calling it, can use the image sensor to see what it is being pointed at and then respond by changing the image it is projecting:

Okay, this isn’t even close to being an ebook story, but when I first heard about it a few days ago I was intrigued. This concept solves many of the problems I have seen when someone’s tried to integrate augmented reality into books.

By placing the projector and the sensor in the same position HideOut avoids the awkwardness and technical problems that can be caused when AR projects depend upon a computer’s webcam (here, here, here).

And as you can see here, it can work quite well with a paper book:

disney hideout

Of course, the cost of all the extra hardware is probably going to remove any chance that this might show up as a book-related product, but this could still become a gaming controller. That future controller could build on the idea of the Sony Wonderbook, which launched last June,  and the Microsoft IllumiRoom.

That last project is basically the same idea as the HideOut, only scaled up to include an entire room (clearly Microsoft and this Disney/CMU research team have been working in the same direction) while the Sony Wonderbook was a much simpler game for the PS3 that let players cast spells from Harry Potter and see the effects on screen.

I don’t know when we’ll see HideOut integrated into a gaming controller, but I do expect it to happen. If it is produced on the same scale as the XBox or the new PlayStation then the cost will drop significantly. It would also be fun to play with, don’t you think?

Disney Research

Fujitsu Now Developing a Virtual Touch Interface for Books

fujitsuRemember back when SF movies used to feature devices with impossibly thin and impossibly flexible displays? That’s what I was thinking of today when I saw the latest project to come out of one of Fujitsu’s labs.

Fujitsu has just unveiled a next-gen user interface which uses a webcam combined with an overhead projector to enable users to interact with real world objects like books, paper, and more.

It’s called the FingerLink Interaction System, and this interface uses the webcam to track a user’s finger or hand movements and then responds to the user by changing the output of the overhead projector.

If the user drags a finger over an image, the system will scan it and upload it. If the user selects a paragraph it will be highlighted by the projector – much like you would see on a touchscreen equipped tablet.

Check out this video for more detail:

As cool as this is, it’s really not anything new.

I know of researchers that have been using tricks like this for some years now as they try to theorize new ways we might interface with electronic devices. The Human Media Lab at Queen’s University in particular has been using this very idea. Here’s one video from 2010 that is largely the same thing (only simpler).

Of course, the HML wasn’t trying to develop the kind of interface that Fujitsu demoed above. Researchers at that lab were always thinking about the devices that might be around the corner.  They were trying to understand how one might use devices like a larger touchscreen equipped table or a bendable mobile display. Yes, these gadgets are practical now, but the HML came up with them at a point when they were still science fiction.

In any case, Fujitsu shows us what is possible if you build gadgets based on the tech you have rather than wait for the tech you want. The FingerLink is much more likely to become a real and useful commercial product than some of the concepts demoed by the Human Media Lab.


New Augmented Reality App Makes Japanese Newspaper into a Game for Kids


The Japanese newspaper Tokyo Shimbun has launched a new app which they hope will attract beginning readers into an aging customer base.

The app, AR News, uses augmented reality to promote certain stories published in the print edition of the daily newspaper. Kids can scan selected articles with an iPhone and the app will show them a "kid-friendly" version with animated characters and graphics, pop-up headlines and explanations about the topics being discussed. The online version of the stories are rewritten so they are easier for beginning readers to understand, with the Kanji characters replaced with the simplified Hiragana alphabet.

Here’s a video demo of the app:

It’s an interesting idea, and I have to say that it’s a better option than the dumbing down of articles that can be found in many US newspapers. But as technically interesting as this may be, I don’t see why the old, analog newspaper needs to be included. Why not just post the news stories online, where everyone is already reading them?

What’s more, the blue graphic used to mark some of the compatible stories is IMO visually distracting. If I were reading the newspaper (I can’t recall the last time that happened) and came across one of these graphics on a story, I would tend to skip it rather than work to decode the text under the graphic. If I really wanted to know about the story I would probably look for it online. In either case, the graphic has cost the newspaper a reader.



New App Brings Augmented Reality Ghosts to the iPad in All Their Appropriate Fuzzyness

The kids books publisher Goosebottom Books is planning to release a ghost-themed augmented reality book next month and they’ve just posted a demo on their website.

The book is called Horrible Hauntings, and it’s a cute look at some of the more famous ghost stories like the Headless Horseman, the Amherst Poltergeist, the Flying Dutchman, and Bloody Mary. All in all there are ten historical ghost stories in this book (pity there’s no Myrtle), and they are  brought to life via the tricks of augmented reality.

You’ll need to buy the book and download the iOS or Android app, but once you do you’ll be able to use the rear facing camera to see and interact with the ghosts. There’s even going to be a scene where you can take the role of a poltergeist and send objects (chairs, an umbrella, vases, and chamber pots) flying around the room.According to one of the developers, there are other simple games embedded in the app, including a spot where you can have the skeletons play ball or blow on the sales of the Flying Dutchman, which fill up when you blow on them.

You can check out the demo video at the end, but I don’t think this is going to work out very well. First and foremost, I’m going to have to agree with one of my readers in that I don’t see the point of requiring an iPad as well as the book in order to use the better parts of the book.

But more importantly, this particular AR trick isn’t working well for me because I’m having issues with the camera on my iPad 2. But that’s a sub-1MB camera anyway, so I wouldn’t expect it to do much in the first place. Here’s what I see on my screen:
As you can see, it looks awfully fuzzy. Assuming this is par for image quality on the iPad 2, I really don’t see the point of using it to support augmented reality apps. It’s just not worth it.

Once again augmented reality is little more than a gimmick. I think this app would be better off if the publishers simply dropped any requirement that the book be involved. But then they wouldn’t be able to pitch the app based on the gee-whiz value of augmented reality, and without that I’m not sure anyone would be writing about them today.

Now This is the Google Glass Demo That I Would Want to Buy

When Google unveiled Project Glass earlier this year the concept video showed both far more than what Google Glass could do right now as well as far less than what was conceptually possible. I think the storyboard that I clipped above does a much better job.

Google’s concept showed hardly anything other than a guy getting info, and it did absolutely nothing to augment what he was seeing. What they could have done was show a more complex interaction like the idea conceived by Sam Saxton. He’s a freelance storyboard artist, and last week he posted his idea for a Google Glass concept video. He posted a storyboard, not a video, so you’ll have to use your imagination to translate the sketches (like the 4 above) into the video.

I like it. Sam has a much better idea of what you can do when you put a camera and screen between yourself and the rest of the world. Rather than just show a few paltry bits of information, Sam’s concept has the users changing what they can see via graphical overlays. The users are both interacting with the environment around them while the overlays still improve upon it.

The clippings I excerpted above and below are from a storyboard that focuses on a jogger. As you can see, the jogger is planning her route in her head’s up display. Eleswhere in the storyboard the jogger picks a game to play while jogging (MarioKart, and there’s also a Zombie running game). Sam also has his version of Google Glass blocking billboards with an "ad blocked" message.

That last is perhaps the most useful feature, IMO. Some parts of the real world are plastered with even more ads than the web, and I’d love to be able to use an augmented reality ad blocker. After all, I already block ads online; why not move that into my offline life (what little there is of it).

I didn’t know Sam’s policy on copying his work, so I only clipped the little bit you see above. You can find the rest over at his blog. It’s well worth a look; Sam also has other storyboards that show how augmented reality glasses might help a student or a football fan.

Valiant Comics Takes Augmented Reality Where It’s Never Been Before – Let’s Hope It Doesn’t Go There Again

Do you recall how I like to describe augmented reality in books and ebooks as a gimmick? I found an example today that deserves the ridicule. On Friday BoingBoing posted a video of a new cover from a new comic book series. This series actually launched a couple months back, but the trick cover is so cheesy that no one cared to write about it.

As you can see in the following video,  Manowar #1 comes with a QR code and a a dotted outline that is about the size and shape of an iPhone. You’re supposed to scan the QR code and be directed to a website where a video clip will be played full screen on the iPhone.

I think it’s cheesy, but apparently I am the only one who does.

There was a link in the BoingBoing comments section that led to a collection of similar "iPhone over mouth" augmented reality tricks. This has been used a lot, and some weren’t bad. For example, one guy used it in his resume; he couldn’t get his foot in the door but he could still get the recruiters to listen. Now that was clever.

And a couple of the other examples look like they’re ads. I’m not sure what they’re saying, but do you know what? It’s still cheesy.  If you want to do a video, link to it. Don’t try some Clutch Cargo kind of gimmickry.

Sony Launches Faux eBook Wonderbook Game for the PS3

So it looks like I was mostly correct in my prediction about the Sony "Storyteller" rumor last week; at E3 last night Sony announced a new type of augmented reality game for the Playstation. They’re calling it the Wonderbook, and the first title is going to be a Harry Potter tie-in called Wonderbook: Book of Spells.

Sidenote: this must be why Sony partnered with Pottermore last year, not for any ebook trick.

It turns out that this is only vaguely related to ebooks (but the tech is still interesting). The game is going to feature original content written by JK Rowling as well as new and fun activities that make use of the motion-sensitive controller on the PS3. Players will compete in the game by learning how to cast spells, conjure up visions, and learn the secrets of wizardry – just like a student at Hogwarts.

It’s not clear from the announcement, but the Wonderbook appears to be a new accessory for the PS3. Going by the video posted here, it looks to be some kind of anchor point for the augmented reality aspects of the game. The PS3’s camera will track your movements (and that of the controller) and, depending on you casting the spell correctly, the PS3 will show on your TV screen the results of the spell you just cast, and it will make it look like the magic is right in front of the player. The image above shows what the kid would see on the screen.

If I’m correct then Book of Spells is  game which will use Wonderbook and is likely sold separately.

J.K. Rowling said, “ This is an extraordinary device that offers a reading experience like no other.” No shit. This isn’t a reading experience; it’s a game. While many confuse the 2, I can see the difference.

The game and accessory are due out in November 2012, and I’d expect them to have rousing success. Kids will love it and parents will buy it because they’ll think it’s reading. That’s fine, but like enhanced ebooks this likely won’t improve literacy. Eye-hand coordination, on the other hand, will go through the roof.