Out of the Text Box

The Menu You Could Not Read

You are at a restaurant abroad, staring at a menu in a language you do not speak. You hold up your phone, point the camera at the page, and ask your assistant what is on it. In a few seconds it reads the menu back to you, translates it, and flags which dishes are vegetarian. You never typed a word.

A year or two ago, the only way to use these tools was to type. They lived inside a text box, and if you wanted help with a photo or a sound, you were out of luck. Now you can show your assistant a picture, play it a recording, or just talk to it out loud.

These tools have gone multimodal, working with pictures, sound, and video now, not just text. It is one of the biggest changes in how they work, and most people are already using it without knowing the word.

What "Multimodal" Means

The word looks technical and means something simple. Multi is many. A modal is a mode, a way of taking in or putting out information. Text is one mode. A picture is another. So is sound, and so is video.

An early chatbot had one mode: text in, text out. A multimodal assistant has several. You can feed it words, images, audio, or video, and it can answer back in more than one of those too, including a spoken voice.

Think of it as the difference between a pen pal and a person sitting across from you. The pen pal only reads your letters. The person can look at what you are holding, hear your tone, and watch what you point at. The tool went from reading your notes to using its senses.

What It Lets You Do Now

This matters because it removes the hardest part of asking for help, describing the thing.

Point your camera at a strange rash and ask what it might be. Show it the error light on your dishwasher and ask what is wrong. Snap a photo of your kid's math homework and have it walk through the problem. Record a meeting and ask for a summary. Hold a conversation out loud while you cook, hands never touching the keyboard.

For anyone who found typing out a careful question tedious, or who is not a strong typist to begin with, this is what finally makes the tool reachable. You stopped having to translate the world into text first. You can just show it.

The Catch

All that convenience has a flip side. For your assistant to see and hear, it has to be given something to see and hear, and that something leaves your device.

The photo of your home, the sound of your voice, the video of your child, the document on your desk: when you share it, you are sending it to a company's servers to be processed. The same camera that reads a menu can read the papers behind it. The same microphone that takes a voice command can pick up the rest of the room.

None of this is a reason to avoid the feature, only a reason to treat your camera and microphone the way you would around any stranger in the room, useful to point at the menu, not left running over the whole table.

Safe Harbor: Three Things You Can Do This Week

• Try showing instead of typing. Next time you would struggle to describe something, point your camera at it or just ask out loud. This is the feature most people are paying for and not using.
• Mind what is in the frame. Before you snap a photo for your assistant, glance at what else is caught in it: documents, faces, screens, addresses. Crop or clear the background the way you would before posting a picture in public.
• Close the mic when you are done. Voice mode is handy, but a live microphone does not know when your question ends. Turn it off between uses so it is not listening to the room by habit.

Next week: How to spot something that was made by AI.