Three weeks ago Amazon had their annual AWS re:Invent event, where new products were launched. I wanted a quick way to test some new products, and picked something practical: Take a photo, understand what’s in the photo using Amazon Rekognition, find a relevant haiku on Reddit /r/haiku, and read it out loud using Amazon Polly. Why not.
Here’s a video demonstrating Haiku Camera:
I wrote a simple web application in ASP.Net MVC, using the AWS .Net SDK. They work quickly, and the NuGet packages for all new services were ready for use shortly after the announcement.
Amazon Rekognition
Rekognition’s DetectLabels can receive a photo and return a list of things it identified in the photo.
The API is straightforward and works nicely. While developing I tested it mostly with photos I took on my trip two years ago, and overall it did quite well, finding relevant keywords for every photo.
The code is fairly short (not handling the case of >5MB images):
public async Task<List<string>> DescribeImage(System.Drawing.Image image)
{
using (var stream = new MemoryStream())
{
image.Save(stream, ImageFormat.Jpeg);
stream.Seek(0, SeekOrigin.Begin);
var rekognition = new AmazonRekognitionClient();
var content = await rekognition.DetectLabelsAsync(
new DetectLabelsRequest
{
MaxLabels = 5,
Image = new Amazon.Rekognition.Model.Image
{
Bytes = stream,
},
});
return content.Labels.Select(l => l.Name).ToList();
}
}
Here are a few good examples using the Rekognition demo page (AWS account required):
As I’ve said, I had a specific goal – I wanted to find a haiku related to the photo. I limited the API to five keywords, assuming that would be enough, and focusing only on the most relevant features in the photo. I could have also used the confidence to remove less likely labels, but I chose not to bother.
After using enough test photos, I noticed I was getting a lot of the same keywords, for example:
It was obvious:
Unfortunately, as you probably already know, people
All of these photos have the keywords People, Person, and Human. Arguably, it is only useful in the photo of dancing people, where the people are really the subject. I search for a haiku based on all keywords, and people are a popular subject among haiku poets. People are spamming my results, and I keep getting the same haiku.
Additionally, the photos of the lion statue and Disneyland have exactly the same labels, adding Art, Sculpture, and Statue.
Confidence is not enough
Besides correctness, another issue is percision and specificness. Consider the results ["bird", "penguin"], or ["food", "sushi"]. It is clear to us, people-person-humans, that Bird ⊃ Penguin, and Food ⊃ Sushi – but how can we choose the more specific word automatically? If I’m using a black-box product like Amazon Rekognition, I probably don’t have the resources to build my own corpus. Furthermore, this data is clearly already contained in Rekognition – but it is not exposed in any way. This is a complementary service – had I used Rekognition to tag many photos and build an index, and wanted to answer questions like “find all photos with [Bird]”, I would not have had this problem. There is difficulty in choosing the best label when describing the content of a single photo.
I did not test it, but AWS has a new chatbot service called Amazon Lex (currently at limited preview) – maybe a chatbot service can help and choose the more specific words.
Technically correct
What’s in this photo?
Image source getyourguide.com.
Ask a few people, and chances are they’ll say The Eiffel Tower, or Paris.
Rekognition gives us {"Architecture", "Tower"}. Now, if a person gave you this answer there are two options: either they’re 3 and know what architecture is, or they have a superb sense of humor. And that’s really the problem: without proper names, Rekognition is being a smart-ass.
Rekognition – conclusion
Rekognition works well enough, and is easy to use. Its face recognition capabilities seem much more advanced than its modest image labeling and content identification.
What could be better:
- A simple way to represent hierarchy between keywords.
- Besides confidence, expose the relevance of the keyword. As reference, this is how Elasticsearch handles relevance:
Inverse document frequency: How often does each term appear in the index? The more often, the less relevant. Terms that appear in many documents have a lower weight than more-uncommon terms
This just makes sense. If you have
"People"in 60% of the photos (not a real number), you can be confident that a photo will have people in it, but it would not be very interesting. - Relevance can also be influenced by the composition of the photo: is there a huge tower in the center and a few small people below it? Maybe the tower is more interesting. It would be nice if the API returned relative areas of the labels, or bounding-boxes where possible (also useful for cropping).
- A few proper names would be nice.
It is fair to mention that this is a first version, and even a small update can make much more useful for my use case.
Finding a haiku
This was the easy part. There are many collections of haiku on the interent. I chose Reddit /r/haiku because Reddit has a simple RSS API for content, a build-in search engine, and a huge variety of crazy creative haiku.
var keywords = String.Join(" OR ", subject.Select(s => $"({s})"));
var url = $"https://www.reddit.com/r/haiku/search.rss?q=title%3A({Uri.EscapeUriString(keywords)})&restrict_sr=on&sort=relevance&t=all";
var all = XDocument.Load(url).Descendants().Where(e => e.Name.LocalName == "title")
.Select(e => e.Value).Where(h => h?.Count('/'.Equals) == 2).ToList();
// retrun a random haiku.
Using the keywords I build a URL for the search API. The filter looks like "title:((Dog) OR (Bear) OR (Giant Panda))".
If I used these haiku publicly or commercially (!?), I would have also check the license, and would have extracted the author and link to the haiku.
Amazon Polly
Another new service is Amazon Polly, which is a voice synthesizer: it accepts text and returns a spoken recording of that text.
public async Task CreateMp3(string text, string targetMp3FilePath)
{
var polly = new AmazonPollyClient();
var speak = await polly.SynthesizeSpeechAsync(new SynthesizeSpeechRequest
{
OutputFormat = OutputFormat.Mp3,
Text = text,
VoiceId = "Joanna",
});
using (var fileStream = File.Create(targetMp3FilePath))
{
speak.AudioStream.CopyTo(fileStream);
}
}
Again, the code is simple, and Polly’s SynthesizeSpeech works easily. Polly has a variety of English voices and accents, including British, American, Australian, Welsh, and Indian, and I pick a random English one each time.
I am not the target audience, but I found the American and British voices to be clearer and of higher quality than the other English voices. (I mean in Amazon Polly, of course. Not in general.)
A minor challenge was to get the punctuation right. The poems in Reddit are presented as three lines separated by slashes and usually spaces. For example:
of all the virtues / patience is the single most / irritating one
This is not a format that is too suitable for Polly. Polly pretty much ignores the slash when it is surrounded by spaces, but reads a verbal “slash” when it is not surrounded by spaces (“warrior/poet”).
Haiku tend to have minimal punctuation, but in cases the punctuation is there I prefer to keep it. When there is no punctuation at the end of a line I add commas and periods:
of all the virtues,
patience is the single most,
irritating one.
This is not ideal, but renders nicely in Polly. I add newlines just for presentation. Newlines are ignored by Polly, as far as I could see.
Polly is obviously not meant for reading haiku, but in this case its quirks are part of the charm.
I did not try SSML at all – it probably requires better semantic understanding than I have.
Other
This is fairly little code to achieve what I wanted – understand what’s in a photo, find a haiku, and play it. I wrapped it all in a small mobile-friendly web page:
- Turns out an
<input type="file">field can trigger the mobile camera. That’s neat. - I used CSS for all styling and animations. CSS can do a lot.
- There is just a little JavaScript. Most of it deals with updating elements and CSS classes – it is not as fun as using an MVVM/MVC framework.
See also
- Source code can be found on GitHub: https://github.com/kobi/HaikuCamera
- Using
DetectLabelsfrom a C# AWS Lambda function
Thanks!
Kobi's Blog 
ohhh its amazing.. can u share the document how to make this??