Newshot

App+1 | NodLink: Turn AirPods into a mouse and control your computer with a bit of brainpower

AI summary

Move the cursor by turning your head, click by tilting it, and speak to give commands. No camera, and no extra device to wear. Read the full article

Translation(model translation, not the source's own text)

Article title: App+1 | NodLink: Turn AirPods Into a Mouse and Control Your Computer With Your Head

Body: Matrix is SSPAI's writing community, where we advocate sharing real product experiences and practical insights and thoughts. From time to time, we select the best articles from Matrix to showcase the most authentic experiences and perspectives from users.

The article represents the author's personal views, and SSPAI has only made slight edits to the title and formatting.

Your AirPods Have Actually Always Known Which Way Your Head Is Facing

Friends who use AirPods have probably all tried spatial audio: when you turn your head, the sound stays in its original direction, as if a speaker were placed in front of you.

This works because AirPods contain a gyroscope and an accelerometer that constantly read the orientation of your head. This data was originally used only for spatial audio, but starting with macOS 14, Apple opened up an interface called CMHeadphoneMotionManager, which third-party apps can also read.

The NodLink I built is simple: since the earbuds know which way your head is facing, let the cursor follow along.

The motion sensors in AirPods read head orientation 50 times per second
The motion sensors in AirPods read head orientation 50 times per second

I hope it can help friends with limited hand mobility, such as people with physical disabilities or those who are bedridden for long periods. Using a computer is too natural for most people, but for them, moving a mouse or clicking a button can be very difficult.

Why Not Use a Camera

Apple's system actually includes a built-in "Head Tracking" accessibility feature that uses the front camera to watch your face. But the camera approach has several unavoidable problems: you need light, you need to face the screen directly, and your face has to be in the frame. Dedicated head-controlled mice often cost over a thousand dollars and require wearing an additional reflective sticker or sensor.

NodLink only reads the sensors in AirPods and never turns on the camera. So in theory, lying down, lying on your side, or having the lights off makes no difference. For bedridden people, I feel this point is quite critical.

Lying down, on your side, with the lights off—all fine
Lying down, on your side, with the lights off—all fine

What It Can Do

Turn your head left or right and the cursor moves left or right; look up or down and the cursor moves up or down. After calibration, your head angle is directly converted into a position on the screen—wherever you look, the cursor is there.

Turning your head is pointing: wherever the extension of your head orientation lands, the cursor is there
Turning your head is pointing: wherever the extension of your head orientation lands, the cursor is there

When looking at things, people rarely tilt their heads, so head tilting is especially suitable as a "button" and does not conflict with cursor movement:

  • Tilt your head left without returning to center: press and hold, allowing dragging; return to center to release

At the same time, all of these can be changed. For example, you can change left head tilt to "hold Fn for voice input" to type by speaking directly; when scrolling videos, left head tilt can also mean "next one."

One action, multiple mappings
One action, multiple mappings

If you find head tilting inconvenient, you can also just speak: "click," "double click," "right click," "swipe up," "swipe down," "copy," "paste," "undo," "close window"... Both Chinese and English work, and phrases like "click" and "command tab" are recognized too. Speech recognition uses the system's built-in on-device recognition.

There is also a small detail here: as soon as you start speaking, the cursor freezes, and after recognition finishes it executes at the frozen position. Otherwise, during those few tenths of a second of recognition, a slight head movement would make the cursor drift off. When buttons are too small and hard to aim at with your head, you can also say "labels": near where your head is pointing, clickable buttons are marked with numbers 1-9, and saying the corresponding number clicks it; when executing a click, the cursor also automatically snaps to a nearby clickable button and draws a box around it to show you which one is selected.

For situations where you want to turn your head to look elsewhere without the cursor running off, say "pause," and say "continue" to resume. Slowly tilting your head, or tilting while turning your head, will not be treated as a click. Press Esc, or touch a real mouse or keyboard, and control immediately returns to your hands.

What It's Like in Practice

The images below are all screenshots from my demo video, recorded while wearing real AirPods and without my hands touching the keyboard or mouse. The small window in the lower right corner shows my own movements.

On the left side of the console is the head tilt angle, with left-right head turning at the bottom and up-down nodding on the right. Once you actually put them on, you'll find that Apple's sensor is really sensitive and the values are very stable; the entire head control system is based on its data.

Real shot: the console displays head angles in three directions in real time
Real shot: the console displays head angles in three directions in real time

Below is a little character I drew myself. Milk Frog doesn't move on its own; every one of its actions comes from your head: nod and it smiles, and the faster you nod, the faster it smiles; when you stop, it stops too; when you turn your head, it also makes a move.

Possessing Milk Frog: nod and it smiles
Possessing Milk Frog: nod and it smiles

I also made four practice scenarios in the console, all of which are the most common things people do on a computer, so you can get familiar with the controls here first.

  • Close a pop-up. Point your head at the button and say "click." The moment you speak, the cursor stops first, and after recognition it clicks at the stopped position.
Close a pop-up: point your head at the button and say "click"
Close a pop-up: point your head at the button and say "click"
  • Scroll short videos. Tilt your head left for the next one, say "double click" to like, and tilt your head right to open the comments.
Real shot: scrolling short videos
Real shot: scrolling short videos
  • Reply to messages. Tilt your head left without returning to center, which equals holding Fn, and speak using the input method; return to center to release, and the text goes on screen. Finally, say "click" to send.
Reply to messages: tilt your head to hold and speak, return to center to enter text
Reply to messages: tilt your head to hold and speak, return to center to enter text
  • Order takeout. Chain the previous methods together: use voice to select dishes and choose a delivery time, tilt your head to dictate a note, and finally place the order.
Real shot: ordering takeout, with the note "leave at the door"
Real shot: ordering takeout, with the note "leave at the door"

To be honest, voice recognition occasionally mishears things. While recording the video, it once heard my "click" as "weather." When that happens, nothing happens, and you just say it again.

Beyond the practice scenarios, I also tried it on the real Douyin website: tilt your head left to switch to the next video, without using my hands at all. (Other people's videos in the image have been blurred.)

The real Douyin website: tilt your head to switch to the next video
The real Douyin website: tilt your head to switch to the next video

For debugging convenience and also for looks, I made a 3D scene for the console. In third-person view, you can see a little figure wearing headphones, and which screen the extension of the head orientation lands on; switch to first-person, and it is your own desktop view.

Third person: see clearly where your head is pointing
Third person: see clearly where your head is pointing
First person: your desk, your cursor
First person: your desk, your cursor

A Bit About the Principles Behind It

If you're interested in the implementation, here are a few things I find relatively interesting.

  • Calibration uses the tip of your nose, not your eyes. After putting on the earbuds, first align the tip of your nose with the point in the center of the screen and hold for 2 seconds to set "straight ahead." For more accuracy, you can do another path calibration of about 30 seconds: first look at the center and the four corners, then use the tip of your nose to follow the cursor along the four edges twice. You must use the tip of your nose here, because AirPods measure head orientation, not gaze direction; just moving your eyes does nothing.
  • Anti-shake. A human head can never stay completely still, and if the cursor follows it directly, it will keep shaking. NodLink uses a filter called One Euro: when the head is still, it suppresses shaking strongly, and when you turn your head quickly, it tries not to slow you down.
  • Drift correction. AirPods have no compass, so the left-right direction can only be accumulated little by little from the gyroscope, and over time it slowly drifts; eventually you might have to turn your head to one side to see the center of the screen. Fortunately, when using a computer, people are generally facing the screen, so NodLink quietly and slowly pulls it back in the background. If it really drifts too much, just say "recenter" or press C.

Technically, the Mac side is a small program written in Swift (I call it the "Headphone Bridge"), responsible for reading earbud data, listening for voice commands, and controlling the system mouse; calibration, filtering, gesture recognition, and the 3D scene are all in a web console written with Node.js and three.js.

How to Install

You need a Mac running macOS 14 or later and a pair of AirPods that support head tracking (3rd generation and later, Pro, Max). The specific installation steps are all written in the GitHub README; just follow them.

After installation, follow the guide at the top of the page: connect the earbuds → set straight ahead → start head control. The permissions and data used are all processed only on your local machine. If you don't have AirPods, you can directly open the online version and use the mouse to simulate head movement to get a feel for it first.

NodLink currently only supports Mac, and installation is a bit of a hurdle: you need to open a terminal and install Node.js, which is not very friendly to people unfamiliar with computers. In the future I also want to make it an app that can be installed with a double click. Another issue is that voice recognition occasionally mishears things, as mentioned above; if it mishears, just say it again.

Actually, what I most want to make is a phone. Phone screens are small, buttons are dense, and it’s the device many people use most while lying down. But iOS doesn’t allow one app to control other apps, and pure software can’t do it. So I plan to add a very cheap ESP32 board: the app on the iPhone reads AirPods head-tracking data and sends it to the ESP32; the ESP32 then connects back to the same iPhone as a Bluetooth mouse and uses the system’s AssistiveTouch to move the cursor and click. But that’s all for later.

NodLink is open source under the MIT license. You’re welcome to try it out and to file issues. If you use a head-controlled mouse or other assistive devices yourself, or have family or friends around you who need one, I’d especially like to hear your feedback. Tell me what feels awkward and which actions still can’t be done.

Follow 少数派小红书 and enjoy a wonderful digital life 🍃
Practical, easy-to-use genuine software, presented to you by 少数派 🚀

The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.

Published by 少数派
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗