Apple introduces the Music Understanding framework

Apple has introduced a new AI-powered tool called the Music Understanding framework that operates on-device.

https://www.youtube.com/watch?v=L7ylGu1vFdc

The framework can analyse audio across six dimensions:

• Key:

  The musical key of the music.

• Rhythm:

  The pulse of the music driven by individual beats. The frameworks builds beats into bars, then phrases, segments and ultimately song sections. Sections are synonymous with a verse, chorus or bridge.

• Structure:

  The sections, segments and phrases of a song.

• Pace:

  Although a song may have a consistent tempo, different sections may feel slower or faster. The framework calls this pace.

• Instrument activity:

  The instruments in the recording, individually mapped over time and strength of activity.

• Loudness:

  The framework analyses the variable loudness of the music over time.

The Music Understanding framework shows how Artificial Intelligence can be used to build the next generation of music and video editing software. It unlocks features such as syncing videos to beats or loudness, or organising a music library by tempo or key.