Thursday, September 4, 2008

You Should Build It: Instrument-Recognition for iTunesR

It seems like we all have massive amounts of music files, and our collections are only growing. To listen to these digital files, we use iTunes or Media Player, or any number of other music applications. The more I use these music applications, it becomes more and more obvious that we, as music listeners, have become database administrators (of a sort). —We are constantly adding to libraries, organizing, creating mixes, playlists and smart playlists, renaming and modifying genres, and on and on. I spend way too much time grooming the files and tweaking the data to make my searches and playlists more predictable and effective. What if we had more control? What if we had the ability to search by one or many instruments, without having to write keywords into our music files? If the computer could determine it programmatically, we could get so much better control of our music and not be restricted by genre definitions that may have been incorrectly assigned by someone else and pulled in when you ripped the songs from CD.

The enhancement I want to see, whether it is to the player, or in a plugin, it doesn't matter. I want the ability to search the library by musical instrument (for example: "find: Piano"). I don't want to have to tag the files by listening to every file... I just have too many files to even consider that.
I'm convinced that the right brilliant person could write code that would evaluate a file's sound signature for the pattern that is "piano", and then return files that matched the signature. Preferences could be set to control how many or how few results you wanted. Or if you wanted to find songs that are only piano, or if it is okay to return songs that show some amount of piano (for example) for a defined amount (for example: Tolerance: Piano: Return files with ___% of song is Piano. Higher percentage will return songs with less of other instruments.)
Find: Drum -or- Find: Percussion
Perhaps as part of a smart play list, you could define: "music-type is/is not piano"

I know that iTunes has a Beat Per Minute feature. This might be a cousin to that. Not many people use the BPM feature because iTunes doesn't determine BPM when it rips files, so you need to get that info elsewhere. So ideally the music player itself would be identifying the music type. If someone like Apple would incorporate this into iTunes, it could be thoroughly integrated into smart searches and be the most beneficial. As an add-on, it would be necessary to have the app run a scan and then add keywords to matching files

The biggest challenge with this concept is that song files, are "mixed down" onto one or two tracks and not on separate tracks, as they were likely recorded. When multiple tracks are used, piano (for example) can be totally isolated, and identified. when you mush all the instruments together, it gets more and more difficult for a computer to identify one sound from another. It would be relatively easy to find a classical music piano piece that features a solo pianist, and becomes exponentially more difficult with each additional instrument/any complexity added to any piece of music.