music-moderation-

This repository contains all the code developed for the Music Moderation track as part of the fellowship cohort of Jan-Apr 2022.

The project aims to analyze an end user’s piano performance by :

  1. Matching the notes played against the corresponding sheet music image file.
  2. Extracting and plotting various features like ‘Pitch’,’Velocity’,’Tempo’ from the audio file.

Table of Contents

Setup and Requirements

For a list of required python packages see the requirements.txt or just install them all at once using pip.

pip install -r requirements.txt

Pipeline

image

The above diagram captures the main steps and the flow of the process:

  1. Automatic Music Transcription (AMT) - We make use of google’s magenta transformer. More information at https://magenta.tensorflow.org/transcription-with-transformers

  2. Optical Music Recognition (OMR) - We make use of an existing open source repo - https://github.com/BreezeWhite/oemer
  3. music21 Libary - A Python-based toolkit for computer-aided musicology. More info at : https://web.mit.edu/music21/

Dataset

Dataset from flowkey website was scraped using a utility at https://github.com/MatthiasLienhard/flowkey_dl

Dataset is located at link and contains around 65 piano performances in form of:

The dataset only contains Intermdediate, Advanced & Pro level songs.

The dataset can be downloaded like below:

  1. !pip install –upgrade –no-cache-dir gdown

  2. %cd /content/gdrive/MyDrive/BaselineExperiments

  3. !gdown –id 1V4tZryiQgTUjgUSGhhvqd62BYaE9qaj4

  4. !unzip BaselineDataset-20220222T143509Z-001.zip

Please note : While scraping data from flowkey, video recordings of songs were converted to ‘wav’ format using an online tool.

Tutorials

If you want to just get familiar with the underlying functions & concepts, check out or tutorials. So far we provide the following tutorials as ipython notebooks:

  1. Librosa_STFT_Intro
  2. AccuracyConfusionMatrix Tutorial
  3. SystematicAnomalyAddition&Detection Tutorial
  4. Compare images Tutorial
  5. Embedding Tutorial

Usage

There are 2 modes to run the script:

1) Note accuracy comparison b/w Audio recording & corresponding sheet Image

First, transcribe the raw audio file using notebook “MagentaTranscription.ipynb”

Then, run the following command to execute the script and compare the Sheet Image PNG file and transcribed MIDI (obtained above).

python .\AudioSheetAnalyzer.py -t “C:\transcribedFiles” -x “C:\xmlFiles” -s “C:\sheetimageFiles”

It takes 2 mandatory parameters as input:

  1. -t (transfilepath) i.e directory where transcribed files are located.
  2. -s (sheetfilepath) i.e. directory where sheet image files are located.

Following parameter is optional:

  1. -x (xmlfilepath) i.e. directory where to generate intermediate musicxml file.

Sample Output

image

2) Audio recording analysis

One can pass raw audio recording of piano music in form of ‘wav’ file and the script finds out the following:

i. Beat Analysis - Approximation of tempo as well as the beat onset indices.

ii. Chromagram - Consists of 12 bins representing the 12 distinct semitones (or chroma) of the musical octave.

iii. Key finder - Analyzes the key that a song is in, i.e. F major or C# minor, using the Krumhansl-Schmuckler key-finding algorithm.

iv. Velocity plot - Uses auboio library to plot velocity graph. The key velocity is an integer between 0 and 127, which depicts the intensity of the sound

Sample Output

image

Beatgraph image

Chromagram image

Velocity plot image