@lachlanjcIMA @ NYU

CC – Week 12: Automation – Project

This week, we’re using Google’s Teachable Machine with ml5.js to use ~machine learning~ to identify if you’re a human, a phone, or a human holding a phone 👀 & speak the result aloud!

(Note: requires Chrome or another browser that supports the Web Speech API.)

View demo

The model

Plum & I trained a simple model using Teaching Machine with pictures of us & our iPhones:

Try on Teachable Machine

Making a web app

With p5, ml5, & the pre-made model, it’s pretty straightforward to show a camera feed & the model’s identification on a website:

let classifier
let imageModelURL = 'https://teachablemachine.withgoogle.com/models/AI5i76oG/'
let video
let flippedVideo
let label = ''
function preload() {
classifier = ml5.imageClassifier(imageModelURL + 'model.json')
}
function setup() {
createCanvas(windowWidth, windowWidth * 0.75)
video = createCapture(VIDEO)
video.size(width, height)
video.hide()
flippedVideo = ml5.flipImage(video)
classifyVideo()
}
function draw() {
background(0)
image(flippedVideo, 0, 0)
noStroke()
fill(100, 100, 100, 200)
rect(0, height - 64, width, 64)
fill(255)
textFont('system-ui')
textSize(48)
textAlign(CENTER)
text(label, width / 2, height - 15)
}
function classifyVideo() {
flippedVideo = ml5.flipImage(video)
classifier.classify(flippedVideo, gotResult)
}
function gotResult(error, results) {
if (error) {
console.error(error)
return
}
label = results[0].label
classifyVideo()
}

Adding a voice

But that’s boring! A few weeks ago I used p5.speech for speech recognition, but the library also allows text-to-speech. What if we announced the label aloud every time it changed?

To inititalize the voice, this is all we need, at the top of the JavaScript file:

let voice = new p5.Speech()

Then in the gotResult function:

const newLabel = results[0].label
if (newLabel !== label) {
voice.speak(label)
label = newLabel
}

That’s it!

Lachlan's avatar