If you use an iPhone to send iMessages (or one of many other services …), you’ll probably recognize this:
The dots (a lab-mate calls them “texting indicators,” which seems like a good description) indicate that someone is typing a message. Because I’m writing this down now I’ve Googled for a bit more about how these work, and I’m learning that there may be some questions about when the dots do and don’t appear (see here, though I think I’ve observed behavior that contradicts their claims). Overall, these dots are relatively innocuous. You can’t get push notifications when your crush starts to message you and spends 5 minutes drafting a message that just says, “hey.” But if you (or a computer) were to constantly monitor your top contacts, well, that’d be creepy. So, naturally, we did it … in a very limited and controlled scenario. We did not use this to monitor anyone without their knowledge and permission.
We used Sikuli – a tool that allows you to write scripts that include screenshots of UI elements that can be recognized with computer vision and then automate some basic interactions with your machine (mouse movements, clicks, etc.) – to write a script that monitors for the texting indicators to appear (using iMessage for Macbook), logs the time when they appear, waits for them to disappear, logs the time they disappear, and sends a message to the person who was texting to let them know that you know how long they spent drafting their message.
Here’s that script:
I have a screen-captured video of this, but it includes the phone number of the friend I was messaging and some of the bizarre contents of my desktop that are probably best kept to myself (e.g., my friend Joe’s head photoshopped onto a turkey in a file called “roast.psd” … actually, that one’s pretty good, maybe I should ask him if I can share that more publicly).
After getting the basics up and running, we modified the script so that when my friend started to type, an IOT lightbulb in the lab turned on. There’s also a video of this happening, but this is mostly just a video of us inexplicably laughing our butts off when a light comes on. IOT Rube Goldberg machines may be coming soon.