The ULTIMATE Piano Learning System

[foxicoot]

Hi everyone! I'm Jake. I'm trying to create the ultimate piano learning system. My idea involves:

1. Dual 27-inch ultra thin bezel monitors mounted upsidedown above my Casio keyboard directly above the keys.

2. Synthesia stretched wide enough across both screens so the keys in the app line up perfectly with the keys on my keyboard.

3. Nvidia Surround to achieve the bezel correction and make the keys in the middle of the keyboard appear to flow behind the monitor bezels.

4. An LED piano strip which is fully customizable and syncs with Synthesia.

I'm basically trying to physically recreate the falling note visualizations (from apps like Embers and See Music) I've seen in modern piano performances on YouTube, but in reverse as a piano learning system for people who can't read sheet music and people like myself who are visually impaired. Here's an example of one of those videos:

https://youtu.be/fzBQvKHsfXU?si=gF_QoAigeSrIkEHU

Here's a screenshot from it:



And here's what my piano learning system looks like in it's current form:





The next step is to install the LED piano strip behind the keys which will light up the keys blue and green to show the user which ones to press. So far, even without the LEDs, it works really well! I've been able to (albeit slowly) play through some of my favorite piano songs that I've wanted to learn ever since I was a kid but never could because they're too complex and I can't read sheet music due to my vision impairment. It's been an incredible learning experience so far and quite empowering!

The only thing I can think of which might improve it is different colors for each of the notes which I see has already been requested multiple times in this forum and technically achieved but not yet released? Nicholas, do you plan to include this feature in the next version of Synthesia by chance?

I was thinking it would be helpful visually and in regards to muscle memory to make the white keys the colors of the rainbow (C = red, D = orange, E = yellow, F = green, G = blue, A = purple, B = pink) and make the black keys darker shades of the white keys they follow (C# = dark red, D# = dark orange, F# = dark green, G# = dark blue, A# = dark purple). What do you guys think about that color coordination system? Would it be helpful for you or no?

Thank you so much to Nicholas for creating Synthesia and to all the members of this forum who contributed to it's development! I am eternally grateful

[Nicholas]

This is a cool project. It reminds me of my old giant-touchscreen setup on a monitor arm hovering over the keyboard. The bezels were big and it didn't cover as many keys, but there is almost no substitute for having the notes fall directly to the real, physical keys of the piano.

Nicholas, do you plan to include this feature in the next version of Synthesia by chance?

This has gotten a little closer, but it's still not quite on what I'd call the "near term" list, sorry.

For the LED strip there is pretty good control/information being emitted by Synthesia these days though. I believe some of the community solutions for LED strips have been able to integrate the per-finger information into their lights.

[foxicoot]

Hey Nicholas!

I agree, there is no substitute for having the notes fall directly onto the actual keys. If it's off by even a little bit, I make a lot of mistakes.

I'm glad to hear note color customization is in the works! That would certainly be a great feature to add not only for enhancing learning but also for improving aesthetics for those who want different color themes.

It's funny that you mentioned using a projector. That's what I used for the first prototype as like a proof of concept. It's not ideal though because the room lights have to be off to be able to see the falling notes clearly and mounting the projector on the ceiling and getting it to line up perfectly with the keys is not easy...

Have you considered creating an AR app which would work with Apple Vision and other AR glasses? I think that would be the key to taking this music reading system out into the real world for solo piano performances, jam sessions and gigs.

Oh I have one technical question! Sometimes when playing through a MIDI file the instrument will change halfway through. Usually from one type of piano to another. Is there a way to prevent this from happening within Synthesia? Or is there a way to make it use only the instrument I select in my Casio keyboard?

Thanks again for creating this awesome app!

[Nicholas]

Have you considered creating an AR app which would work with Apple Vision and other AR glasses?

It's a really cool idea, to be sure. But with a one-person development team, each feature really has to make a big impact to the majority of users to be worthwhile. Until the install base for AR hardware is a little less niche, I'm afraid it wouldn't make sense to prioritize over quite a few of the other things on the list.

Sometimes when playing through a MIDI file the instrument will change halfway through.

Did you mean a regular, intended instrument change? Or is this more like a bug where the instrument is changing unexpectedly?

If it's the former, there are a couple workarounds. Just changing the instrument to your preference (step #2 of this guide) is probably the easiest. The brute-force option is to disable Settings --> Advanced --> Instrument Change Messages. But that is probably more aggressive than you're looking for.

[foxicoot]

Did you mean a regular, intended instrument change? Or is this more like a bug where the instrument is changing unexpectedly?

It always happens at the same point in a MIDI file both when I listen to it and when I try to play the melody so I'm assuming it's an intentional instrument change within the MIDI file. It sounds like a honky tonk and the volume is increased. I tried disabling Instrument Change Messages but that didn't work. I checked if there are multiple instruments in the file but it lists only one: acoustic grand piano. Should I open the MIDI file in Cakewalk Sonar, see what's going on in there and remove the instrument changes if there are any? Or do you have any other ideas?

There's one other thing I really need your help with... I'm having lots of problems with the sustain pedal when I use the Learn the Melody function. Before when I connected my Casio keyboard directly to my Windows 10 laptop the sustain pedal occasionally wouldn't work properly and it would get stuck on certain notes (or so it seemed). And now my keyboard is connected to the Piano LED strip which is connected to my laptop via rtpMIDI/WiFi but it's not registering the sustain pedal at all I think. It was working before but then it just suddenly stopped working. In order to advance in the song, I must quickly tap each of the keys as the notes fall down. If I hold down a key, it will not advance. Is there something I'm doing wrong? Or is this a known bug? I've tried disabling Pedal Changes but to no avail. What should I do? I'll try resetting the Piano LED app, resetting Synthesia, and resetting my laptop to see if that works.

Is it possible to connect my keyboard directly to my laptop and then connect the Piano LED strip to my laptop with rtpMIDI/WiFi? Or do I need to connect my keyboard to the Piano LED strip for it to work? I'm wondering if this would fix the problem, but I'm not sure if this is possible...

[foxicoot]

Is it possible to connect my keyboard directly to my laptop and then connect the Piano LED strip to my laptop with rtpMIDI/WiFi? Or do I need to connect my keyboard to the Piano LED strip for it to work? I'm wondering if this would fix the problem, but I'm not sure if this is possible...

This is indeed possible and it solved the problem of the sustain pedal not registering in Learn the Melody mode. However, I am still having the problem where the sustain pedal is getting stuck on certain notes (but not often). Is there a fix for this? Is it possible to reduce the input sensitivity for the falling notes in Learn to Play mode? Like where it allows more mistakes or it allows keys to be pressed a few milliseconds earlier?

FYI the instrument change problem seems to have been resolved by playing with the inputs/outputs after connecting my keyboard directly to my laptop via MIDI cable.

[foxicoot]

FYI the instrument change problem seems to have been resolved by playing with the inputs/outputs after connecting my keyboard directly to my laptop via MIDI cable.

Actually, disabling all outputs is what fixed this problem. That's surprising because I thought disabling the output for my Casio keyboard would result in no sound at all but it just plays whatever instrument I select on my Casio keyboard through its speakers.

[Nicholas]

Is it possible to reduce the input sensitivity for the falling notes in Learn to Play mode? Like where it allows more mistakes or it allows keys to be pressed a few milliseconds earlier?

Hmm, usually you are allowed to resume a note (say, if your finger slipped off) without a pause in melody practice.

Try heading over to Settings --> Gameplay and check under the "Melody Practice" heading there. The "Incorrect Notes ... Are Ignored" option might be a band-aid fix, but it sounds like your input might be getting doubled-up somehow. (The "Continue When" option there might also help relax some of the melody practice requirements.)

[foxicoot]

That worked! I'm not sure if it was "Continue When Each Note Has Been Pressed" or "Incorrect Notes Are Ignored" but with these settings melody practice is super smooth now. Thank you

I have one small feature request... If the ability to assign a color to each note is not feasible or would take too long to program or just isn't a priority, then please just add one more color - a 7th color - for the instrument tracks. Then I can manually put all the C's and C# in a track in Cakewalk and make them all red and dark red in Synthesia, make all the D's and D# orange and dark orange, and so on. Then perhaps pink, brown, gray or white for B. My personal preference would be brown. After using Synthesia with my piano learning system for the past few months, I'm certain that color-coding would significantly improve my ability to sightread. I would be very grateful if you could implement this feature request.

Another idea I had which would improve sightreading is an alteration of Next-Note Helper Dots. It would be great if you could somehow indicate the next keys which need to be pressed as opposed to the current keys which need to be pressed which is what it shows now. That would help reduce the amount of time I spend looking up and checking the next notes which are falling. I was thinking it might work if there were vertical lines below the falling notes which fade from 0% opacity to 100% opacity as they get closer to the bottom. Or perhaps a dot on the piano roll keyboard which starts small and gets bigger as the falling note reaches the bottom. What do you think? Have you thought about implementing a feature like this before?