Google: The future of search is Now

Google Now's cards try to predict what you need to know before you search. They serve up information related to previous searches, your calendar, and more. (Credit: CNET screenshots)

Google needs a new tagline: The future of search is Now.

 

It wouldn't be a stretch, given the huge bet Google is making that it can create intelligent digital assistants for billions of people by putting Google's computer brain to work for you via Google Now. The service, which Google this week expanded to iOS users, is about far more than one-upping Siri in the battle for digital assistants. Google is angling to maintain its top position in search as people leave the desktop and search on mobile and wearable devices.

Google Now, arguably not the most compelling name, makes the point: Google wants to tell you what you need to know "now," quickly and accurately. It works by turning natural language queries -- speaking to computer as if to another human -- into precise answers delivered from Google's servers.

At this stage of Google Now, the main interface is "cards," virtual boxes with information on traffic, weather, sports, stocks, public transit, flights, events, shipments, appointments, and so on. You can ask questions, such as "What time does the San Francisco Giants game start?" or "What will the weather in New York be like next week?," and Now offers an info card and audio response when appropriate.

Where Google Now becomes most interesting, and useful, is when it does a mind meld with user data gleaned from your mobile devices, Google services and other, non-Google apps. For example, Now can detect restaurant reservations from your Gmail account, and automatically send you an alert along with with directions from wherever you are. Google Now can detect that you check news and scores on the New York Yankees, for example, and then automatically present the latest scores and news updates.

 Google Now is a key part of the Google Glass experience. (Credit: Google)

Basically, the more Google Now contextual data knows about you, the better it can serve as your digital assistant. 

Google Now is still in its infancy, and the results are hit and miss. But the king of search is mustering its top engineering talent for this initiative to defend its corporate crown jewel, which accounts for more than two-thirds of searches in the U.S. The vast majority of those searches are keyword oriented, returning a list of links.

The field is getting crowded. Besides Apple's Siri, less known upstarts like Sherpa and Donna, also would like to have the job as your digital companion. And specialized digital companions, or assistants, are being designed for cars, the xBox, appliances, and other devices.

The 'Star Trek' computer

The origins of Google Now go back to "Star Trek", as Google search chief Amit Singhal tells the story. In a blog post from 2012, Singhal wrote:

Larry Page once described the perfect search engine as understanding exactly what you mean and giving you back exactly what you want. It's very much like the computer I dreamt about as a child growing up in India, glued to our black-and-white TV for every episode of Star Trek. I imagined a future where a starship computer would be able to answer any question I might ask, instantly. Today, we're closer to that dream than I ever thought possible during my working life.

Google Now is the information cards and pleasant voice responding to questions and anticipating needs like any worthy assistant, but there is a lot going on behind the scenes to produce the illusion of a digital human interfacing with a real human. 

Google has teams focused on speech recognition, language modeling, and creating a computer representation of everything that Google knows, called the Knowledge Graph.

Tamar Yehoshua, director of product management for Google Search.

(Credit: Steve Jennings/Getty Images)

Tamar Yehoshua, director of product management for Google search, says that Google's "Star Trek" dream is still in its infancy. "It takes a tremendous amount of compute power to understand natural language speech, convert it to entities, find answers and then convert text to speech," she said. "This is the beginning stage of showing what we can do."

Google has made significant progress on the initial part of the communicating with an intelligent digital assistant, understanding what users are saying to the machines in the cloud.

"We have had a mini-revolution, based on deep learning, a set of technologies that look like the old neural networks from the 1990s that researchers hoped would turn into a way to create brains, machine robots that became sentient and took over world," said Vincent Vanhoucke, Google's technology lead for acoustic modeling of speech. 

Autonomous robots haven't taken over world, but Google deep learning, which behave like a set of neurons in the human brain connected in a dense mesh and exchanging data -- to acoustic modeling, taking the raw waveforms of speech and determining what the phonemes, such as an "a" or "p," sound like in any speaker environment and accent. Then a language model strings together the phonemes into words and sentences, all probabilistically, Vanhoucke explained.

One of the major breakthroughs is acoustic modeling was using GPUs (Graphical Processing Unit) to train systems, Vanhoucke said. "Neural nets have to pass data quickly and at high density. A GPU has one big shared memory and can pull all the neurons in memory, and parallelize things very well. What took a year to train now only takes three weeks, so we can run more experiments on larger number of machines and train very large networks. 

"The shift from Android before Jelly Bean and after in terms of the accuracy of voice recognition improved by 15 to 30 percent, depending on the language." 

Yet complex conversations, and maintaining context, present problems.

Yehoshua offered this example: "I might be an SF Giants fan having a conversation with any device near me," he said. "I am asking what is happening in the Giants game, who is pitching and the time of the game tomorrow, as well as asking to record the game to my DVR and remind me about the game. To solve this, we have to integrate a whole number of pieces together. It's a hard problem but also extremely exciting."

The 1 percent solution

Google's speech recognition and language modeling is making rapid improvements, but understanding specific meanings remains the biggest challenge. That's where the Knowledge Graph comes in. 

Knowledge Graph feeds Google Now with data about topics, people, events, and other kinds of information, to construct answers. It has more than 570 million entities and 18 billion facts about connections between them, by Google's count.

When Google Now receives a query, it turns the raw speech data into entities the computer understands and then comes up with an answer by matching that with what is in the Knowledge Graph. So when you ask, "How did the Giants do?," the Knowledge Graph will know you're referring to the baseball team -- and not some other giant -- based on your search history. Then, Now displays an information card and reads out the score from the previous night's game. 

However, Knowledge Graph today represents just a small fraction of the entities and relationships languages generate. 

"Knowledge Graph has good coverage of people, places, things, and events, but there is plenty it doesn't know about. We are at 1 percent," said John Giannandrea, director of engineering for Knowledge Graph. "But we are not trying to be like a person. We are trying to be both dumber and smarter. It's a tool that gives you data, context, and better understanding of a problem, but you are still making the decision."

Google performs hundreds of millions of searches per day, providing raw material feeds into the Knowledge Graph. "Every single day 16 percent of queries are new," said Yehoshua. "People have new combinations of what they are searching for all the time. We need to extract what are the entities we can understand. It's a continual process." 

Where Google's Knowledge Graph lives. (Credit: Google)

With more data, deep learning, faster and more vast processing capacity, Google's "Star Trek" computer, and others, will eventually come to life. But it will take a very long time to get your jokes.

"There is a lot that is implied by our understanding of the world and we have to teach the system from the bottom up. We have to have an understanding of analogy, irony, illusion, and all those human things. Computer history suggests this will be a game of inches, rather than a quantum leap, but the rate of progress will accelerate," Giannandrea said.