Mark Zuckerberg Shows Off Jarvis-Inspired AI Home Assistant

As the founder and CEO of one of the world’s biggest and most successful technology companies, Facebook’s Mark Zuckerberg is a man who likes a challenge.

In January this year, he revealed his 2016 aim of building a fully-functional, artificial intelligence (AI) system to help “run my home and help me with my work.” Just under 12 months later and he has achieved his goal.

After around 100 hours of work over the course of the year, Zuckerberg has spent this week showing off his AI-powered personal assistant inspired by Jarvis, the virtual assistant designed by Tony Stark in Marvel’s Iron Man.

Jarvis comes to life

“So far this year, I’ve built a simple AI that I can talk to on my phone and computer, that can control my home, including lights, temperature, appliances, music and security, that learns my tastes and patterns, that can learn new words and concepts, and that can even entertain Max,” he wrote in a blog post.

“It uses several artificial intelligence techniques, including natural language processing, speech recognition, face recognition, and reinforcement learning, written in Python, PHP and Objective C.”

The first issue he encountered was connecting all of the different systems in his house, which he described as being “much more complicated than I expected”. As the systems are based on different languages and protocols, Zuckerberg had to first write code that would connect all of the disparate systems together.

“We use a Crestron system with our lights, thermostat and doors, a Sonos system with Spotify for music, a Samsung TV, a Nest cam for Max, and of course my work is connected to Facebook’s systems,” he explained. “I had to reverse engineer APIs for some of these to even get to the point where I could issue a command from my computer to turn the lights on or get a song to play.”

“Further, most appliances aren’t even connected to the internet yet. It’s possible to control some of these using internet-connected power switches that let you turn the power on and off remotely. But often that isn’t enough.” Some appliances actually required hardware modifications in order for them to work, such as a toaster from the 1950’s which he fitted with a “connected switch” to enable it to automatically start toasting when the power comes on.

He continues: “For assistants like Jarvis to be able to control everything in homes for more people, we need more devices to be connected and the industry needs to develop common APIs and standards for the devices to talk to each other.”

Zuckerberg also explains the subtle nuances involved with natural language requests for actions such as playing music, as well as how he incorporated vision and face recognition so the system can identify which room of the house people are in or recognise who someone is when they knock at the front door.

So, what’s next? “In the longer term, I’d like to explore teaching Jarvis how to learn new skills itself rather than me having to teach it how to perform specific tasks. If I spent another year on this challenge, I’d focus more on learning how learning works,” said Facebook’s fonder. 

Zuckerberg would also like to find ways to make this sort of system “available to the world,” something he was unable to do with this version as it is “too tightly tied” to his home, appliances and network.

“Building Jarvis was an interesting intellectual challenge,” he concludes. “This year I thought I’d learn about AI, and I also learned about home automation and Facebook’s internal technology too. That’s what’s so interesting about these challenges.”

Quiz: How much do you know about the world of AI?