Synthesia 11 Release Estimate

[JimNYC]

After many years of saying there are still many years to go on Synthesia 11, all of a sudden it is looking like there is a good chance that the entirety of beautifully-engraved MusicXML-based sheet music might only take a few more months of integration work instead of more years of from-scratch development.

That's excellent news!! In the past year and a half, I still have yet to find something works as smoothly as Synthesia for custom songs.

If you need any help testing, please don't hesitate to reach out!!!

[JimNYC]

I haven't tested it yet, but I know the library has extensive support for arbitrarily printing text (lyrics, tempo directions, TAB notation, tuplet beat counts, etc.) in the music. So, even if it's something that doesn't exist yet, it will be easier to add it there than to add it to Synthesia.

The code is right on GitHub and the author seems to welcome outside contributions, so it's something I could help maintain, add features to, and fix bugs. (That's probably the most attractive part of the idea. If it were a closed library, I would be stuck with whichever bugs were in there and might have to tell users "sorry, I can't do anything about it". For such an important part of Synthesia, I would never put myself or its users in that position. But because there's ready access to fix things, it should work nicely.)

It looks like fingering is already in Lomse from a previous release. From the change log of version 0.29.0 (current version is 0.30.0 from September):

Version 0.29.0 (11/Mar/2022)
More music notation supported:
Fingering
Pedal marks
All accidental signs
Non-standard key signatures
Full support for cue notes

Improvements in MusicXML support:
Enable edition in MusicXML
Import <staff-lines> elements
Import <defaults> element
Import rest placement on the staff
Import <fingering> elements
Support MusicXML files encoded in UTF-16

[Nicholas]

That's nice to hear a confirmation.

There will still be the extra step of connecting Synthesia's auxiliary finger hint information to the renderer. Otherwise it would only be able to show hints stored in the MusicXML file itself and not those you add via the in-app interface or via metadata, etc. But I don't anticipate that being too tricky.

Well... maybe. There's an interesting correspondence problem between the falling note blocks and the sheet music anytime you have a proper repeat. There will be two (or more) note blocks that represent the same note in the sheet music... so what should be shown in the sheet music if you assign different finger hints to each block?

It's not even that pathological of a question: if it's the last note in the repeated section before choosing between different endings, you may actually want to use a different finger the second time around to lead into the new section that follows. Hmm. I suppose having something in the falling notes that becomes slightly inexpressible in the sheet music isn't the end of the world. I could probably choose whatever is the easiest answer (the hint on the first note block "win" for the sheet music view, etc.)

[JimNYC]

Does this even exist as a construct in actual sheet music? If not, then I would just take the path of least resistance and assign it the first one (as well as having fingering as an option to display or not). If someone really wanted to express different fingering like your scenario in sheet music, they could always collapse the repeats in music editing software and duplicate the sections instead of repeats. Then it would be easy to assign it different fingering for each block.

[Electrode]

I know it's been a while since this post was last updated, but you could do what usually happens in sheet music when alternate fingers are provided. Alternate fingerings are either provided (depending on publisher/edition) by using slash notation or brackets/parentheses.

So, you could express alternate fingering by using 2/3 or 2(3), for example, for a main fingering choice of 2 with the alternate being 3. Of course, this only works for individual changes. For a group of notes with alternate fingering, things might get a little more complicated. That might require something like this, perhaps:

[Nicholas]

That's interesting. I don't recall seeing the slash/parenthesis notation before. In Behind Bars, finger substitution is shown (on p.309) with a dash between the two fingers... but this isn't really finger substitution either. There isn't any mention there of alternate fingerings.

(I like the footnote on that page, hehe.)

[JimNYC]

After many years of saying there are still many years to go on Synthesia 11, all of a sudden it is looking like there is a good chance that the entirety of beautifully-engraved MusicXML-based sheet music might only take a few more months of integration work instead of more years of from-scratch development.

I recently stumbled upon the LenMus project, created by Cecilio Salmerón over even more years than Synthesia. He has (incredibly graciously) open-sourced the sheet-rendering part of his project as a library called Lomse. It's in C++, the license is compatible with Synthesia, and the dependencies are (rather coincidentally) all things that Synthesia already has integrated into its codebase. It uses the same musical font that Synthesia has been using for several years and I've even seen half a dozen references to Behind Bars in the code, so it appears we've even been using the same reference book to make decisions.

Any update on the MusicXML sheet music integration with Loomse? Looking at the development release graph, it doesn't look like there has been any change in the last few months. Is the sheet music integration still looking promising?

[Nicholas]

There has been a delay for sure, sorry. I was pulled in another direction for half a year and I'm just starting to discover the backlog that has been piling up here in the meantime.

In my time away, I thought about Lomse a lot. I suspect it would still be the most expedient way to get competent sheet music into Synthesia. By a pretty significant margin, too. But from a philosophical (and even practical software delivery) standpoint, the "library" is larger than all of Synthesia combined.

Software continues to eat the world, app sizes double every couple years, the hardware gets faster while the actual experience of using apps steadily declines and this doesn't seem to alarm most people... but it's something that I spend a lot of time thinking about.

Most of the Lomse integration work will be in severing many tendrils from the kraken so that it can fit neatly inside Synthesia's code base. How many unforeseen roadblocks will crop up during that process is anybody's guess. If I had a crystal ball, this would be an easier decision.

The urge to do the Right Thing™ and make it myself from scratch is strong, but you guys have been waiting over a decade at this point and the guilt has mounted to almost the breaking point.

[JimNYC]

Glad to see you are back.

While I understand the urge to do it from scratch allows for complete control, there is no doubt that MusicXML rendering is a huge undertaking judging by the state of other apps & tools. One only needs to read through the discord & GitHub for OSMD to see all the various items that come up just to build a renderer. Dev teams with multiple people seem to constantly be working through issues.

Given that Lomse is open source and is in the right language/structure to integrate, it seems like you get the best of both worlds: a starting point which has the nuts and bolts figured out, plus the ability to alter/customize it as needed. Synthesia is so great with it's ability to play along with each hand, but the sheet music is not really usable as is for practicing something unfamiliar. Any improvement there would be welcomed, especially if it isn't going to require years to implement.

I don't think the file size itself matters to anyone, but performance certainly is a factor. Maybe there could even be an option during installation which allows one to choose whether to support MusicXML rendering and if not selected, it is something similar to how it is today in terms of filesize and functionality.

[Nicholas]

You're probably right.

I only lament that I've gotten through about half of the "hard" parts: the actual primitive rendering (like, drawing individual pixels) in Synthesia's code is very nice and super fast. I paid special attention to making each line in a staff uniform, having the corners of bar lines match up with the staff, not having anything look blurry, and all these other little details that every other notation app (including the major ones) often get wrong... but using Lomse for rendering means not getting to use any of that.

The scarier part is that while Synthesia can now reasonably competently load MusicXML files, Lomse will have to do it again (under the hood) and then the glue code will have to try to reconcile any differences between the results. I can't know for sure, but that feels like it might be a rabbit hole of integration work that takes a lot of effort with nothing really useful being added.

That last bit is the risky unknown that has been keeping me undecided.

[JimNYC]

On the flip side, the path to doing MusicXML rendering from scratch doesn't have to be all or nothing if it's not too daunting. It can be incremental and support a core set of items initially (particularly notes on correct staff, stem direction, ties, etc.). Fingering notation would also be really useful. Other more complicated items in the XML could be supported over time. Anything closer to the real notation would be an improvement users could benefit from.

[Chompy]

I just have a question - when will Synthesia 11 ever come out? Sounds like Synthesia 11 is going to be very big, and Synthesia 11 has been worked on since 2020.

[Nicholas]

... support a core set of items initially (particularly notes on correct staff, stem direction, ties, etc.).

Drawing ties (and slurs more generally) is really the last piece of "new tech" that Synthesia's own sheet rendering component would need before an incremental approach was possible. Otherwise, blindly drawing the note-heads described in the MusicXML would make a tied note appear as separate notes, which would be even less usable.

(I suppose an honorable mention might go to showing more than two beams. The code to draw beams exists, but there's a little bit of work before it could show finer than 16th notes.)

... and Synthesia 11 has been worked on since 2020.

It's more embarrassing than that: the main topic was posted in early 2016, and I'm pretty sure I'd already been talking about "better sheet music" for more than a few years at that point. Searching the forum, it looks like I had started talking seriously about eventually supporting MusicXML files all the way back in 2009.

[JimNYC]

So, it sounds like Lomse is out and it's going to be built from the ground up if I had to guess. The posts over a year ago had so much promise so I only hope this is feasible. From my perspective, having notes appear on the correct staff would be a huge improvement from the current musicxml rendering.

[Nicholas]

So, it sounds like Lomse is out...

Not necessarily. That bit about ties being the last of the new drawing tech required has been true for at least two years now. (So it wasn't new information or anything.)

Just getting things on the correct staff would be relatively "easy" now in Synthesia's code, too. That could be one of the first small steps, if need be. That said, I do still plan to give Lomse a shot and see if it "just works". If there are any major blockers that come up, I might not go down that path, but from the quick test I was able to whip up a year ago, it seemed feasible.

[JimNYC]

Just wondering if you had any indication on the viability of Lomse "just working". Don't mean to bug you, but if' it's any consolation, no other app (I'm aware of) does what Synthesia does with practice mode and playing the other hand, so I'm hopeful we can get some basic sheet music improvements like the notes in the right staff in place.

[Nicholas]

I haven't gone any farther down the Lomse investigation path yet. Synthesia has been a bit of lumbering beast to get back up and running as a project now that I'm back from my nearly one-year hiatus. At this point most of the fires have been put out, I'm as caught up on email as I'm going to get, and I'm back to mostly-regular development.

In terms of missing underlying drawing "technology" in Synthesia's own sheet renderer before it could be on par with what Lomse is able to generate, I count three things:

1. Text in arbitrary fonts.
2. Beams that span more than two notes.
3. Nice, swoopy generated ties/slurs.

For #1, Synthesia can draw text just fine. The "hard" part here is actually finding the font on the device. If the MusicXML file says to write something in italics... which font is that on macOS 10.12? What about on an Android 6.0 tablet? Or Android 11? Where is that file located? The last thing I want to have to do is ship a handful of full Unicode fonts with the app. Lomse's own support for this today seems a bit ad hoc and would need to be extended to the platforms that Synthesia supports that Lomse doesn't yet. The right answer is probably something closer to using fontconfig.

#2 is a smallish one. It has less to do with drawing beams (which is already specialized to be ultra-fast with nice anti-aliasing) and more to do with lengthening stems to match those beams. Really, I'm not too worried about this but it's in the list for completeness.

Progress Update!

#3 was a large technology hurdle. Until it wasn't! This has been one of my bigger projects since I've been back. I learned about splines, curvature, curve continuity, and all sorts of other fun math bits. This slur was generated by Synthesia's own sheet renderer and manually placed by hand as a tech demo:

slurTest.png (33.71 KiB) Viewed 7054 times

I think it looks quite nice.

Since then I've been working to get the MIDI-to-sheet engine to finally generate the extra note heads for notes that last longer than the remaining duration of a measure. This tie was generated completely automatically from a MIDI file and simply screen-grabbed from the app:

firstRealTie.png (21.75 KiB) Viewed 7054 times

There is still some work to be done before I can say the sheet music supports ties in general (notably for ties that continue to the next line), but the majority of the hard part is complete.

#3 was the big, scary one of the three. And now... it's not. So the value proposition of incorporating Lomse has become a little weaker. It's still an open question but Synthesia's own sheet music renderer is now also one step closer to "just working".

[jimhenry]

Congratulations on slaying #3. Your examples look very nice!

[JimNYC]

Awesome to hear about the updates and glad you were able to work out some lingering issues. There's certainly no need to use Loomse if it can be rendered directly and you don't have to reinvent the wheel to handle the MusicXML. I think the biggest win will be just getting Synthesia to render the sheet music closer to the MusicXML definition, but not necessarily completely perfectly all at once. Right now, it's somewhat unusable for certain pieces when notes aren't showing on the correct staff or stems are in the wrong direction, especially when learning a new piece.