The ULTIMATE Piano Learning System

Have questions? Just saying hello? This is the place.
No explicit, hateful, or hurtful language. Nothing illegal.
Post Reply
foxicoot
Posts: 5

Post by foxicoot »

Hi everyone! I'm Jake. I'm trying to create the ultimate piano learning system. My idea involves:

1. Dual 27-inch ultra thin bezel monitors mounted upsidedown above my Casio keyboard directly above the keys.

2. Synthesia stretched wide enough across both screens so the keys in the app line up perfectly with the keys on my keyboard.

3. Nvidia Surround to achieve the bezel correction and make the keys in the middle of the keyboard appear to flow behind the monitor bezels.

4. An LED piano strip which is fully customizable and syncs with Synthesia.

I'm basically trying to physically recreate the falling note visualizations (from apps like Embers and See Music) I've seen in modern piano performances on YouTube, but in reverse as a piano learning system for people who can't read sheet music and people like myself who are visually impaired. Here's an example of one of those videos:

https://youtu.be/fzBQvKHsfXU?si=gF_QoAigeSrIkEHU

Here's a screenshot from it:

Image

And here's what my piano learning system looks like in it's current form:

Image

Image

The next step is to install the LED piano strip behind the keys which will light up the keys blue and green to show the user which ones to press. So far, even without the LEDs, it works really well! I've been able to (albeit slowly) play through some of my favorite piano songs that I've wanted to learn ever since I was a kid but never could because they're too complex and I can't read sheet music due to my vision impairment. It's been an incredible learning experience so far and quite empowering!

The only thing I can think of which might improve it is different colors for each of the notes which I see has already been requested multiple times in this forum and technically achieved but not yet released? Nicholas, do you plan to include this feature in the next version of Synthesia by chance?

I was thinking it would be helpful visually and in regards to muscle memory to make the white keys the colors of the rainbow (C = red, D = orange, E = yellow, F = green, G = blue, A = purple, B = pink) and make the black keys darker shades of the white keys they follow (C# = dark red, D# = dark orange, F# = dark green, G# = dark blue, A# = dark purple). What do you guys think about that color coordination system? Would it be helpful for you or no?

Thank you so much to Nicholas for creating Synthesia and to all the members of this forum who contributed to it's development! I am eternally grateful 🙏
Nicholas
Posts: 13169

Post by Nicholas »

This is a cool project. It reminds me of my old giant-touchscreen setup on a monitor arm hovering over the keyboard. The bezels were big and it didn't cover as many keys, but there is almost no substitute for having the notes fall directly to the real, physical keys of the piano.
foxicoot wrote: 05-09-24 3:31 pmNicholas, do you plan to include this feature in the next version of Synthesia by chance?
This has gotten a little closer, but it's still not quite on what I'd call the "near term" list, sorry.

For the LED strip there is pretty good control/information being emitted by Synthesia these days though. I believe some of the community solutions for LED strips have been able to integrate the per-finger information into their lights.
foxicoot
Posts: 5

Post by foxicoot »

Hey Nicholas!

I agree, there is no substitute for having the notes fall directly onto the actual keys. If it's off by even a little bit, I make a lot of mistakes.

I'm glad to hear note color customization is in the works! That would certainly be a great feature to add not only for enhancing learning but also for improving aesthetics for those who want different color themes.

It's funny that you mentioned using a projector. That's what I used for the first prototype as like a proof of concept. It's not ideal though because the room lights have to be off to be able to see the falling notes clearly and mounting the projector on the ceiling and getting it to line up perfectly with the keys is not easy...

Have you considered creating an AR app which would work with Apple Vision and other AR glasses? I think that would be the key to taking this music reading system out into the real world for solo piano performances, jam sessions and gigs.

Oh I have one technical question! Sometimes when playing through a MIDI file the instrument will change halfway through. Usually from one type of piano to another. Is there a way to prevent this from happening within Synthesia? Or is there a way to make it use only the instrument I select in my Casio keyboard?

Thanks again for creating this awesome app! 🙏
Nicholas
Posts: 13169

Post by Nicholas »

foxicoot wrote: 05-11-24 10:00 amHave you considered creating an AR app which would work with Apple Vision and other AR glasses?
It's a really cool idea, to be sure. But with a one-person development team, each feature really has to make a big impact to the majority of users to be worthwhile. Until the install base for AR hardware is a little less niche, I'm afraid it wouldn't make sense to prioritize over quite a few of the other things on the list.
foxicoot wrote: 05-11-24 10:00 amSometimes when playing through a MIDI file the instrument will change halfway through.
Did you mean a regular, intended instrument change? Or is this more like a bug where the instrument is changing unexpectedly?

If it's the former, there are a couple workarounds. Just changing the instrument to your preference (step #2 of this guide) is probably the easiest. The brute-force option is to disable Settings --> Advanced --> Instrument Change Messages. But that is probably more aggressive than you're looking for.
foxicoot
Posts: 5

Post by foxicoot »

Nicholas wrote: 05-14-24 2:41 pm Did you mean a regular, intended instrument change? Or is this more like a bug where the instrument is changing unexpectedly?
It always happens at the same point in a MIDI file both when I listen to it and when I try to play the melody so I'm assuming it's an intentional instrument change within the MIDI file. It sounds like a honky tonk and the volume is increased. I tried disabling Instrument Change Messages but that didn't work. I checked if there are multiple instruments in the file but it lists only one: acoustic grand piano. Should I open the MIDI file in Cakewalk Sonar, see what's going on in there and remove the instrument changes if there are any? Or do you have any other ideas?

There's one other thing I really need your help with... I'm having lots of problems with the sustain pedal when I use the Learn the Melody function. Before when I connected my Casio keyboard directly to my Windows 10 laptop the sustain pedal occasionally wouldn't work properly and it would get stuck on certain notes (or so it seemed). And now my keyboard is connected to the Piano LED strip which is connected to my laptop via rtpMIDI/WiFi but it's not registering the sustain pedal at all I think. It was working before but then it just suddenly stopped working. In order to advance in the song, I must quickly tap each of the keys as the notes fall down. If I hold down a key, it will not advance. Is there something I'm doing wrong? Or is this a known bug? I've tried disabling Pedal Changes but to no avail. What should I do? I'll try resetting the Piano LED app, resetting Synthesia, and resetting my laptop to see if that works.

Is it possible to connect my keyboard directly to my laptop and then connect the Piano LED strip to my laptop with rtpMIDI/WiFi? Or do I need to connect my keyboard to the Piano LED strip for it to work? I'm wondering if this would fix the problem, but I'm not sure if this is possible...
foxicoot
Posts: 5

Post by foxicoot »

foxicoot wrote: 05-23-24 3:43 pm Is it possible to connect my keyboard directly to my laptop and then connect the Piano LED strip to my laptop with rtpMIDI/WiFi? Or do I need to connect my keyboard to the Piano LED strip for it to work? I'm wondering if this would fix the problem, but I'm not sure if this is possible...
This is indeed possible and it solved the problem of the sustain pedal not registering in Learn the Melody mode. However, I am still having the problem where the sustain pedal is getting stuck on certain notes (but not often). Is there a fix for this? Is it possible to reduce the input sensitivity for the falling notes in Learn to Play mode? Like where it allows more mistakes or it allows keys to be pressed a few milliseconds earlier?

FYI the instrument change problem seems to have been resolved by playing with the inputs/outputs after connecting my keyboard directly to my laptop via MIDI cable.
foxicoot
Posts: 5

Post by foxicoot »

foxicoot wrote: 05-23-24 5:44 pm FYI the instrument change problem seems to have been resolved by playing with the inputs/outputs after connecting my keyboard directly to my laptop via MIDI cable.
Actually, disabling all outputs is what fixed this problem. That's surprising because I thought disabling the output for my Casio keyboard would result in no sound at all but it just plays whatever instrument I select on my Casio keyboard through its speakers.
Nicholas
Posts: 13169

Post by Nicholas »

foxicoot wrote: 05-23-24 5:44 pmIs it possible to reduce the input sensitivity for the falling notes in Learn to Play mode? Like where it allows more mistakes or it allows keys to be pressed a few milliseconds earlier?
Hmm, usually you are allowed to resume a note (say, if your finger slipped off) without a pause in melody practice.

Try heading over to Settings --> Gameplay and check under the "Melody Practice" heading there. The "Incorrect Notes ... Are Ignored" option might be a band-aid fix, but it sounds like your input might be getting doubled-up somehow. (The "Continue When" option there might also help relax some of the melody practice requirements.)
Post Reply