In the latest Shaun of the Shed episode, Shaun Preece walks us through the new Be My Eyes app for windows. Focused strictly on the Be My AI features, you can use it to describe image files, webcam photos & the layout and contents of your screen or clipboard.
The Be My Eyes app is now available for Windows, specifically for the AI feature.
Be My Eyes is an app that allows users to create a video call with a sighted volunteer for assistance. It has evolved to include artificial intelligence (AI) functionality called Be My AI, which can provide descriptions of images.
The Windows app offers four main options: describe the screen, describe a picture taken using the built-in camera, describe a picture from file explorer, and describe the clipboard. The AI provides detailed descriptions and can answer follow-up questions. The app is simple to use and can be beneficial for blind or partially-sighted individuals.
Chapters
- [Narrator] "Shaun of the Shed," an AMI original podcast.
- Hello, you beauties, and welcome to another episode of "Shaun of the Shed." Today, I wanna take a look at Be My Eyes. I know, I'm always talking about Be My Eyes on "Shaun of the Shed," but there's a good reason for that. It's an incredibly useful app. Now, if there's some of you out there that don't know what Be My Eyes is, then don't panic. That's absolutely fine. No one knows everything but what it is, a quick rundown of what it is, it's an incredibly popular app that allows you to quickly create a video call with a sighted volunteer. So let's say you've dropped your keys or whatever it may be and you've felt around and you can't find them, just open up Be My Eyes, hit the button to make a video call and a sighted volunteer. There's seven and a half million of them on this app, amazing, will answer. And they can see whatever your rear camera of your smartphone is pointing at. Since that initial use case of sighted human volunteers, it has evolved. And there's the other side of the Be My Eyes app and that is using artificial intelligence. Now with this functionality, it's called Be My AI. Clever. See what they did there? What this allows you to do is to take a picture, so a photograph of anything you like or even any photo that's already on your phone or maybe a an image from an email or a webcam, whatever it may be. You can send it to the Be My Eyes app and this time, instead of using humans, artificial intelligence will take a look at whatever that image is and give you a description of it. It's incredibly useful for going through the camera roll, all those photos in your gallery on your phone, which you have no idea what they are, and just so many other things. It works really well. Now with all this said, if you want to know more about Be My Eyes at least on the smartphone, as I said, I have talked about before on "Shaun of the Shed," the first time using the human volunteer aspect of the app. You can find by searching for "Shaun of the Shed" and possibly Be My Eyes as well, episode five in your favorite podcast app. This was back when I was audio only. And recently here on YouTube, I also showed the AI version or functionality of Be My Eyes. If you search for "Shaun of the Shed" and Be My Eyes in YouTube, you should find that one too. So that's just to give you a background of what Be My Eyes is. But this new rollout today, this new functionality is the Be My Eyes app coming to Windows, to your laptop or your desktop computer. Why would you want that? The only functionality that the Windows versions of Be My Eyes brings is actually the artificial intelligence and there's a good reason to that. You don't really want a human being being able to look at whatever's on your computer screen. There could be privacy concerns, security concerns over that. So that's not available yet. That's not to say it won't happen in the future, but they're gonna need to take some time and thought over how to implement in that. But right now, Be My Eyes for Windows allows you to use AI to describe any visuals on your computer. It's gonna be very, very handy and I'll give you an example of for why we're gonna jump straight into it now. Firstly, let me show you how to get Be My Eyes for Windows. All you need to do is open the Microsoft Store. Now, if you don't know what that is, don't worry again because it's just like the app store on your iPhone or the Play Store on your Android phone. It's just somewhere to get software, but it's built directly into your copy of Windows. So let's open it up. Just hit the Windows key to bring up the start menu. Okay, so let's type store
- [AI] Microsoft Store.
- [Shaun] That's what I want. I'm just gonna hit Enter to open it up.
- [AI] Microsoft Store window.
- [Shaun] Okay, so Microsoft Store is now opened. So first thing I need to do is jump to the search field. To do that, I'm just gonna hit Control and E.
- [AI] In search edit.
- [Shaun] So let's type in Be My Eyes.
- [AI] Be My Eyes, my suggestions.
- and hit Enter. Now we've got a list of results on the screen. To get to them, I'm just gonna hit Tab until I find the Be My Eyes app
- [AI] All departments Tab on one of five, Be My Eyes, three total ratings, average rating of 3.7 out of five stars.
- [Shaun] There it is. Okay, hit Enter on that.
- [AI] Customized, open button, open Be My Eyes.
- And that's it. Now as you heard, the focus said open button because obviously I've already installed Be My Eyes on this computer, but when you come to do this, it'll say something like install or get, I can't actually remember which word it uses, but just hit Space or Enter on that button to download it and install it on your computer. And that's it. How easy was that? We now have Be My Eyes installed on this computer. So let's open it up and see what it can do. Okay, so before we open up the app, let me just say I am using Windows 11 and the screen reader that I'm using today is Narrator simply because it's got a really nice voice that I like and my usual screen reader, NVDA, I thought I'd just take a break, but whatever screen reader you're using or even if you just use magnification, then you'll be able to follow along with this quite easily, I think. I used NVDA, I've used JAWS, and of course right now I'm using Narrator to use Be My Eyes and I've had no problems at all. Anyway, thankfully, just like when we installed Be My Eyes, actually using the app is incredibly simple as well. There really isn't much to it. Currently, there's four options that we can use and we'll jump into those in just a second. But when you open Be My Eyes for the first time, if you've not used it before, then you will get a sign in or sign up screen. Now, if you've used Be My Eyes on your smartphone before, then you will already have an account and you simply sign in with your email and password. If you haven't, then signing up for the Be My Eye service is really easy. There's only a few questions, your name and your email, and a password and you're done. Once you've signed in or signed up, you'll never have to deal with that again. So let's open it up and see what it can do. Again, let's hit the Windows key to bring up the start menu,
- [AI] Start window.
- [Shaun] And let's type Be My Eyes,
- [AI] Be My Eyes app.
- [Shaun] That's what I want. Let's hit Enter to open it up.
- [AI] Be My Eyes window, describe my screen, shortcut Control and Alt and H.
- [Shaun] And there we go. It's open. And as you heard the screen reader focus, immediately jumps to the first tool or the first option in the app, which is describe my screen. And what you'll also hear is that it reads out the keyboard shortcut to activate that function. So we can either hit Space or Enter on this button right now to describe whatever's on my screen. Or we could use the keyboard shortcut which is Control + Alt and H. We're gonna try this out in just one second, but before we do, let's have a quick look around in this app. Let's Tab to the next option.
- [AI] Take picture using built-in camera, shortcut Control and Alt and J button.
- [Shaun] There you go, that's very handy. If you're recording a video like this or getting ready for a Zoom meeting, whatever it may be, you can get a description of what your webcam can see to make sure you're appropriately dressed or looking at the camera, whatever it may be. Let's Tab and see what the next option is.
- [AI] Describe picture from File Explorer, shortcut Control and Alt and K.
- [Shaun] So if you've got a photo saved on your computer somewhere or any image file for that matter, this option allows you to browse your computer for it, open it up and get a description, Tab.
- [AI] Describe clipboard, shortcut Control and Alt and L.
- [Shaun] Now, it's not very often that I copy images to clipboards as a screen reader user and someone who can't actually see the screen anyway. It's very rare that I bother with images at all. But this could be useful. For example, if you're surfing the web, you can right click on any images and copy to clipboard and this option would give you a description or maybe you're going through an email with embedded photos in it. You can Control + C on any of those images and again, use this option to get a description. And that's the four main options we've got. But there are some other things here. So let's Tab on.
- [AI] Be my AI tab item, one of two, level one selected.
- [Shaun] Okay, so the other things we have in this app are tabs. So the one we're in currently, Be My Eyes has all these functions in it, the ones we've just gone through. If I just use the right arrow to switch to the next tab, again, I'm using Narrator. If you're using NVDA or JAWS or something else, then you may use a different command to switch tabs such as Tab itself. But I'm just gonna hit right arrow
- [AI] Settings tab item, two of two, level one
- And the other tab is settings. But to be quite honest, right now, there's only one option in there and that is to log out of your account. But hey, there may be other options a little bit later on. So there you go, that's the Be My Eyes app and as you can see, it's really simple. So let's just try some of this out. The first thing I'm gonna do is get a description of whatever's on my screen. And in fact, let's get a description of the Be My Eyes page in the Microsoft Store because I've still got it open. So I'm gonna Alt + Tab to make sure that's in screenshot or that's in focus.
- [AI] Be my Eyes, two of five videos through Microsoft Store, Microsoft Store window.
- [Shaun] Okay, and now remember those keyboard shortcuts. I'm gonna hit Control + Alt and H to get a description
- [AI] H, Be My Eyes Window, settings. Please wait, Be My AI is writing a message. Question to ask the AI, edit. The image shows a screenshot of the Microsoft Store page for the Be My Eyes app. On the left side, there is a section with the apps icon, which is a blue square with a white sunburst pattern in the center.
- That's it, Control-
- Smiling at the-
- [Shaun] Just to stop it speaking. But as you heard, so much detail there, right? And gives you the description of the screenshots that are on this page. Absolutely fantastic. Now, if I want to get more details or ask another question, all I need to do is simply start typing my question because focus is already in the correct place. So let's ask it, for example, give me a summary of the apps' description.
- [AI] C-R-I-P-T-I-O-N.
- [Shaun] And hit Enter.
- [AI] Please wait, Be My AI is writing a message. The app Be My Eyes for Windows provides our powered descriptions of images and graphics along with an interactive chatbot To enhance computer accessibility, key features include describing the screen, describing local images, and offering detailed descriptions of various onscreen content.
- [Shaun] Fantastic. And that really is the power of Be My Eyes on Windows, but let's call it what it is. Be My AI for Windows because there's no human volunteers, as I mentioned earlier. Now, I've tried this on various screens or various software actually on My Computer. For example, if I'm using Audacity, my audio editing software and I get a description of the screen, if I ask it for more details, it will actually go into real depth about what a button is and what function it has. So it can be even useful for like a instructional manual. It can tell you what a button does and what its use is. Very, very good. When I did the keyboard shortcut Control + Alt and H, to start this description, Be My Eyes opens up. 'Cause remember I was in the Microsoft Store and now I'm in here. So I can just Tab around this description screen.
- [AI] Back button.
- [Shaun] Back will take us back to the main screen,
- [AI] Be My AI tab item, one of two, level one.
- [Shaun] That's the tab we were on earlier. Tab again.
- [AI] Your image, one of four.
- [Shaun] Now, this is the start of the conversation we just had. That image is the image of the Microsoft Store. If I arrow down.
- [AI] The image shows a screenshot of the Microsoft Store page for the Be My Eyes.
- [Shaun] There's the description that I got the first time. So you can go back through and review if you missed anything it said. If I Tab again.
- [AI] Question to ask the AI, edit.
- [Shaun] There's the text fill to ask further questions. And that's pretty much it. Again, very simple, very quick to navigate. If I wanna go back, I could Tab to that back button again, but you can also just use the Alt and left arrow.
- [AI] Describe my screen, shortcut Control and Alt and H.
- And we're back in the main screen of Be My Eyes. How cool is that? Now, that description screen we were just on, which had the answers it was giving me, it's pretty much the same for all of the options. The only difference is that those options are different ways to get photos, to get images to the AI, through the webcam, screenshot of the screen, clipboard, and browsing for a file. So they're all different ways to get images to the AI, but once you've done that, the screens with the description is always the same. So anyway, let's try the next one, which is take a picture with a built-in camera, I think. So this is very handy as well. I used it just before recording this video to make sure that my big, fat face was in frame and I was centered up. It's very handy for things like that and Zoom meetings, whatever it may be. So let's try it out. I'm going to hit Control + Alt + J. That's the keyboard shortcut for this function. But again, if you don't wanna use the keyboard shortcuts or you can't remember them, you can just Alt + Tab until you hear the Be My Eyes app. And then just Tab until you find the button that you want or the option that you want and hit Space to do it. But the keyboard shortcuts are quicker when they work, more on that later on. Anyway, I've also got a slight issue in that, I cannot actually use this option and record the screen at the same time because of the window setup, but it doesn't matter. I've still recorded the audio. You'll be able to hear what's happening. So let's hit Control + Alt and J to take a picture using my webcam. Let me just pose. Here we go.
- [AI] J, Be My Eyes window, taking a photo.
- Okay, good. Taking a photo.
- [AI] Back. Please wait, Be My AI is writing a message. Question to ask the AI, edit. A response, good.
- The picture shows a man with short, light colored hair, wearing a dark green shirt. He is smiling and giving a thumbs up with his right hand. He is wearing large over-ear headphones and is in a room with soundproofing phone panels on the walls, which are black and red in color. There is a microphone on a stand to his right and a guitar leaning against the wall in the background.
- Okay, fantastic. Yeah, absolutely right. I didn't know my shirt was green, but now I do. It's amazing what AI can pick up. Again, we can ask follow on questions. And in the case of the webcam, it may be something like... I'm gonna type it here. Actually, let's just do it. Describe this man more. Is this okay for a video meeting? And hit Enter.
- [AI] Please wait, Be My AI is writing a message. Message sent. Yes, the scene appears to be well-suited for a video meeting. The man is well-lit. The background is professional with soundproofing panels and there are no visible distractions. The presence of the microphone and headphones suggests good audio quality, which is important for clear communication during a video meeting.
- Thanks.
- [AI] The man has short light, blonde hair and a fair complexion. He is smiling warmly, which gives a friendly and approachable appearance. He is wearing a dark green button-up shirt. His build appears to be average and he has a clean-shaven face. The large over-ear headphones he is wearing are black and they are connected to a device out of the frame. His overall demeanor is positive and professional.
- Wow. It's the first time anyone's said that about me. I love AI, but as you can see again, fantastic. Let's check the next one. Let's open a file from My Computer and get a description. Okay, let's be professional. So let's try out the next option, which is to open a file from your computer. I think the option is called describe photo from File Explorer. That makes sense. So I've got a photo saved on my desktop. It's of the Soundcore Frames. Remember those? I think I reviewed those in my very first YouTube shared video. So let's try it out. But this time, instead of using the keyboard shortcut let's do it the long way round if you'd like. I'm gonna Alt + Tab until I find the Be My Eyes app, should actually be just the next one, Alt + Tab.
- [AI] Be My Eyes window. Describe my screen, shortcut Control and Alt and H.
- [Shaun] There we go. Now, let's just Tab.
- [AI] Take picture using built-in camera, shortcut Control and Alt and J button.
- Done it. Next.
- [AI] Describe picture from File Explorer, shortcut Control and Alt and K button.
- [Shaun] That's the one I want. Let's hit Space.
- [AI] Be my AI, open dialogue, file name, combo, edit, collapsed, all turned in.
- [Shaun] So what it happens here is that it opens a standard Windows open file dialogue box. So I'm just gonna navigate around here. I'm not gonna explain what I'm doing, just navigate around until I find a photo. Lemme just Shift + Tab
- [AI] Shaun Shed, navigation pantry, Shaun personal OneDrive.
- [Shaun] Desktop.
- [AI] Items view list, enter table items view. Temp to a four, selected.
- [Shaun] And I think it's called Soundcore.
- [AI] Soundcore Frames.jpg.
- [Shaun] There it is. So I'm just gonna hit Enter on this file now and get a description.
- [AI] Be my Eyes window. Be My Question to ask the AI, edit. The picture shows a pair of black sunglasses placed on a gray texture surface. Behind the sunglasses, there is a can of Diet Coke. To the left of the can-
- Yes, there is. Okay, that's enough of that. Okay. Diet Coke, I know. As you can see or hear, fantastic. Works perfectly well. Now, I know everyone's got an opinion about artificial intelligence at the moment, good or bad, but there's no denying, the level of detail in those descriptions. And the way you can drill down and ask further questions to drag out more information about an image is just so useful to us as blind or vision impaired people. I am loving it. Now, I don't think there's any need for me to demo the final option, which is describe the clipboard because, well, it just works like all the others, to be honest. And as I said, I don't use image in the clipboard very often. But hey, if you do, the option is there. Now, before I go, I quickly want to mention the keyboard shortcuts that I brought up earlier because there is a slight issue with them and that is that they sometimes work or sometimes don't, depending on what other apps you have running. For example, if I have Outlook, the email client running on My Computer, then the keyboard shortcuts won't work. Now, it's not a massive issue because as I said, you can just Alt + Tab to the Be My Eyes app and just navigate and use the buttons. But it is something to be aware of. Now, of course, the Be My Eyes team themselves are aware of this issue. So chances are by the time you're watching this, it may have been fixed, but if not, just so you know, it's not you doing anything wrong, it's just a slight bug in the program. So there you go. Thank you so much. That was Be My Eyes for Windows. I am loving it. It should be called Be My AI. But hey, it is what it is. If you like this video, please hit that like button. If you haven't done already, please subscribe. And if you want some more techie goodness, why not check out "Access Tech Live" or double tap video on YouTube. Thanks, guys. Take care.