Automatic Fingering Prediction 0.1ish

Synthesia is a living project. You can help by sharing your ideas.
Search the forum before posting your idea. :D

No explicit, hateful, or hurtful language. Nothing illegal.
Post Reply
Nicholas
Posts: 13170

Post by Nicholas »

Welcome back. I know exactly what you mean about the GUI parts not being as interesting. ;)

Again, I would like to support this endeavor in any way I can. The metadata editor isn't "note aware" yet, though I won't be able to keep it that way for long. With that in mind, I might be able to bite the bullet and do it now, just to create a little interface where you can click a note block and press a number to assign a finger. (Or, if the track is monophonic, maybe you could just keep pressing numbers and it would auto-advance or something.)

Still, having the notes available so you know the fingering input is still matching up will be reassuring. I guess you can't get into too much trouble if you require the note list and finger list to be the same length, but even transposing a couple numbers could hurt the training results.
vicentefer31
Posts: 899

Post by vicentefer31 »

Nicholas wrote:...just to create a little interface where you can click a note block and press a number to assign a finger...
Something like this?
Picasso: I am always doing that which I cannot do, in order that I may learn how to do it.
Nicholas
Posts: 13170

Post by Nicholas »

Something very much like that. I was thinking right in the metadata editor -- you can really whip together a GUI in C# faster than I've seen in any other language or toolkit -- but yeah, you have the idea. If you make the 1-5 keys on your keyboard shortcuts for clicking the buttons in the finger "palette" there, it would be even closer (but you wouldn't have been able to show that in the video).
vicentefer31
Posts: 899

Post by vicentefer31 »

This is a post I wrote about July-2008.
vicentefer31 wrote:This is my first example:
It´s very simple I have made with free software Anvil. The fingerings are: 1313131313134431313131313....
The song has three track. They are R1 for finger 1 of Right hand, R3 for finger 3 of Right hand and R4 for finger 4 of the right hand.
We can do something like that with Hanon exercices, chords, arpeggios, scales, .... and songs!!!
I hope you enjoy it.
Exercice Blues 1.mid
(965 Bytes) Downloaded 514 times
And this other example with the 5 tracks finger
Pentatonic-Blues-RH.mid
(958 Bytes) Downloaded 360 times
Frost wrote:The new version got delayed but I'm still working. I've finished most of what I was planning, loads midis ...
The idea is: Now Frost software can load midis, but can it save midis?
Example:
A midi with two tracks RH (right hand) and LH (left hand). It was perfect it can save a new midi with 12 tracks:
1) track RH (the same as the original midi)
2) track LH (the same as the original mdi)
3) track R1 (only the notes from the RH track that should be played with finger 1 from the Right Hand)
...
12) track LH5 ((only the notes from the LH track that should be played with finger 5 from the Left Hand)

Synthesia should know this "format file" and then show falling notes for track RH and LH, and for tracks RH1 a text like R1.... and for LH5 a text like L5.

So we can see something like this:
Image
Picasso: I am always doing that which I cannot do, in order that I may learn how to do it.
BenJackson
Posts: 38

Post by BenJackson »

Some complications:

"Real" fingering isn't as tidy as computer programmers might like: You can have one finger hitting multiple keys (rarely, but see the end of Chopin Prelude op 28 #7, eg). You frequently have a key which is hit by one finger and then the sustain is transferred to another finger so the rest of the hand can move (written eg 5,4 or 4,5).
Frost
Posts: 51

Post by Frost »

both can be coded. I can add those features without much effort i.e. multiple keys with a finger can be coded if any other fingering is too hard/impossible and the finger in question can depress those multiple keys. Even, mute sustain changes was possible in the beginning, I disabled it explicitly (i.e. did more work), since maybe the software would suggest those mute changes to most-likely beginners that can't do those maneuvers.

The reason I overlook those is that, they would make things more complicated than necessary. most of the people using this software would not be expert players, and advanced players would not use the software (maybe only take suggestions).
About, "real fingering", if some people did this for a living, I believe we would have a software in a few years that can suggest the best fingering that is possible, much better than a pianist. the "best", because unlike music generation or other artistic endeavor, this is a mechanical problem, and computers are quite good at this, much better than humans at optimizing. the software can take into account personal preferences, body changes etc. to create fully personalized result. But, would the classically taught pianists would use that? I doubt it. At least until a new generation of pianists arrive. Jazz players play more freeform, so they won't use it either. So we are still left with hobbyists and the rare intermediate-advanced player that lacks the fingering in the sheet music and is in a hurry.

In short, those features and other limitations can be overcome with development, but are not the first priority. If synthesia can suggest a suitable enough fingering for your older aunt that plays jingle bells with right hand index finger only, that would be a victory for now. :)
Nicholas
Posts: 13170

Post by Nicholas »

Frost wrote:If synthesia can suggest a suitable enough fingering for your older aunt that plays jingle bells with right hand index finger only, that would be a victory for now. :)
That was awesome. :D

Still, I agree with the "until a new generation of pianists arrive" sentiment. They're out there right now and they're playing games like Rock Band, looking for something equivalent for the instrument their parents are forcing them to play.

Give me another twenty years or so and "classically trained" will mean "instructed by Frost's algorithm"... or is that too ambitious? ;)
BenJackson
Posts: 38

Post by BenJackson »

Frost,

I wasn't referring to your algorithm, but to the metadata. I assume someone out there will want to transcribe fingerings for some piece they are working on. It would be easy to choose the wrong database relation for fingers to keys.

I think people WOULD use a fingering analysis tool if it were available. Especially if it could be customized to the user.
Nicholas
Posts: 13170

Post by Nicholas »

That's reasonable Ben. I think Frost and I are sort of splitting the persistence problem (or, I might even be taking it full-on myself).

At least in the Synthesia case, I can't imagine any restrictions on duplicates showing up for finger numbers. The notion of "overlapping" notes is already fuzzy at best when working with MIDI. For example, if you're not quantized properly, sequential notes might appear to overlap slightly. So, any "no duplicates" policy would have to be pretty lax to begin with.

This will become even more interesting some day when feeding those values into the anatomically correct, 3D, inverse-kinematic model of hands that is attempting to show the correct finger and hand position in real time... but that's a problem for another day. ;)
Frost
Posts: 51

Post by Frost »

I wasn't referring to your algorithm, but to the metadata. I assume someone out there will want to transcribe fingerings for some piece they are working on. It would be easy to choose the wrong database relation for fingers to keys.
oh, sorry :)

yeah, the metadata would be problematic. it should be robust to allow personal fingerings, but be able to cope with problems in midi, easy to programmatically use, and maybe in a human-readable/editable format? It's kind of a "pick any two" problem. I'll let Nicholas deal with it for now :)
TonE
Synthesia Donor
Posts: 1180

Post by TonE »

vicentefer31 wrote:The idea is: Now Frost software can load midis, but can it save midis?
If we

1. have Frosts AFP which can load .mid files and output e.g. fingering information as .txt files (bestly via the command line, again :lol: ) and
2. know how Nicholas will support fingering information in Synthesia, e.g. if and how it will support fingering information directly in the .mid file itself

then we should be able to write Frost's AFP output information easily into the original .mid file (or just any other .mid file if you do not want to change your original .mid files) using Keykit and AutoHotkey, automatically, without any human hand use, even for whole directories and subdirectories all in one shot.
vicentefer31
Posts: 899

Post by vicentefer31 »

No news, bad news
Picasso: I am always doing that which I cannot do, in order that I may learn how to do it.
Nicholas
Posts: 13170

Post by Nicholas »

Oh come on, I disappear for longer stretches of time than this. ;) Perhaps a picture of Frost in his own fashion shirt will surface soon.

But no, I am pretty anxious to hear about more updates myself.
Frost
Posts: 51

Post by Frost »

oops. I'm constantly fighting for deadlines, hence still not found much time. Also, I restarted my piano lessons, the teacher is not happy with my hands and tries to relax them into a mush. So, no more time reading about fingering, rather, studying fingering :)

On the good side, polyphonic midis. On the bad side, I need to totally change the representation for polyphonic midis. Also, there should be a way for selecting left hand as well, which is not trivial as it sounds (the code was hacked together quickly, so not much architecture below. I'll need to refactor it).

I'm not sure about the actual fingering performance, feedback would be nice. Maybe I can clean the monophonic midi version next time I find some time and release it as an early test version, so you can load and play around midis. No promises though.
vicentefer31
Posts: 899

Post by vicentefer31 »

Is it possible to add fingerings in Synthesia for monophonic midis with just one track?
Example:
hanon_1_RH.mid
hanon_1_RH.txt

With this, Synthesia Users can do midis for learning Hanon exercises, scales, arpegios,..for left and right hand and this could be a nice step to help Frost with his software. I think it has to be really easy to add but useful until we can have in the future the "Manual Note Fingerings "

Papa Smurf, what do you think about this?
Dibujo.png
Dibujo.png (80.33 KiB) Viewed 42593 times
Edit:
Grouchy Smurf: I hate Manual Note Fingerings. Don't vote for this feature here!!!
Picasso: I am always doing that which I cannot do, in order that I may learn how to do it.
Nicholas
Posts: 13170

Post by Nicholas »

vicentefer31 wrote:I think it has to be really easy to add but useful until we can have in the future the "Manual Note Fingerings
Hmm... I think it *is* manual note fingerings.

For me, automatic note fingerings would be like the integration of something like Frost's tool right into the game. Until then, you're using some outside (manual) source of data to annotate the notes.
TonE
Synthesia Donor
Posts: 1180

Post by TonE »

vicentefer31 wrote:Is it possible to add fingerings in Synthesia for monophonic midis with just one track?
Example:
hanon_1_RH.mid
hanon_1_RH.txt

With this, Synthesia Users can do midis for learning Hanon exercises, scales, arpegios,..for left and right hand and this could be a nice step to help Frost with his software. I think it has to be really easy to add but useful until we can have in the future the "Manual Note Fingerings "
This is imo very easy to achieve. Nicholas has only need to support e.g. "Lyrics" events in the midi files, then having a pair of .mid and .txt with all the fingering information for each note in .mid, e.g. for the Hanon exercises, would allow us, in my case using Keykit, to add the fingering information from .txt to .mid as "Lyrics". Then Synthesia would show those "Lyrics", which would be from the set of {1,2,3,4,5} in a similar way as in your screenshot.

In short, getting "Lyrics" support on the notes in Synthesia would equal then to manual fingering support. At least for monophonic phrases or patterns this would work.
Nicholas
Posts: 13170

Post by Nicholas »

Hmm... I've got "Lyrics in Sheet Music Display" on the list, but nothing about on the falling notes. I was worried about width concerns. (Even the Fixed-Do names really want to pop outside of the notes.) But, since lyrics are supposed to be single-syllable, maybe it's not so bad.
TonE
Synthesia Donor
Posts: 1180

Post by TonE »

Nicholas wrote:I was worried about width concerns. (Even the Fixed-Do names really want to pop outside of the notes.)
You could limit the "lyrics display on the falling notes" to only a single character or maybe two characters as maximum, if they fit also nicely into a notes width, then we could also display, DO, RE, MI, FA, SO(L) :), LA, SI.

Whenever the syllable would be wider than than two characters, all characters from position >= 2 could be auto-truncated, on the "on the notes display".

Or very stricly allowing only {1,2,3,4,5} on the falling notes as lyrics. Or we would write the lyrics as {L1,L2,L3,L4,L5} for the left hand and {R1,R2,R3,R4,R5} for the right hand and Synthesia would show in both cases only 1,2,3,4,5 but based on L or R showing a different "note head color" on the note.
TonE
Synthesia Donor
Posts: 1180

Post by TonE »

TonE wrote:At least for monophonic phrases or patterns this would work.
Even for polyphonic phrases there is a solution: For example let us assume we have 5 notes which are sounding or even beginning simultaneously, then you would simply write again normally your "lyrics fingering numbers" for each note, starting from the lowest pitch going higher, all lyrics separated by a comma, for that onset time. Or instead a comma we could agree also on another "special delimiter" for that situation. Keykit handles simultaneous notes also according to this strategy, starting from the lowest pitch going higher.
Post Reply