10.8 preview r5658-r5686 - MusicXML Loading

Post by **Nicholas** » 09-24-21 12:08 pm

SYNTHESIA 10.8.5658
Download from the pink box above.

CHANGES SINCE r5621

Experimental MusicXML file loading!
Removed the confusing/incorrect usage of the word "Omni".
Duplicate/old entries should no longer appear in the Recent Songs list after editing a file.
Fixed a bug when deleting Free Play recorded tracks.

This is Synthesia's first step into a larger world. Right now it parses the bare-essentials required to interpret the file as something like MIDI. So the sheet music will be no better than usual (i.e., awful). But everything* should sound correct.

Things that do NOT work correctly:

Segno and coda. (Discussion here.)
MusicXML files using a top-level <score-timewise> element. I haven't been able to find one in the wild and they are generally unsupported in other apps as well. This should be a non-issue.
Hairpins (crescendos, etc.) are currently applied to all notes in a part (which can contain multiple voices and staffs), even though they can technically be made to refer to only one staff.
All "text directions" (any printed words like "cresc." or "poco rit." near the staff) are ignored.
Grace notes are completely omitted.
Pick-up measures aren't detected yet, so every subsequent measure is skewed by the number of beats in the pickup measure. (This will be fixed in the next build. I only noticed it this morning as I was preparing the preview build and didn't want to stop.)

Call to action: Please try to load MusicXML files and check to see that they sound correct (that is: the falling notes should appear how you'd expect). If they don't, post about it here or send the files to support@synthesiagame.com. Mention the measure number where things went awry.

* There are many, many possibilities inside these files (scroll here for fun pictures) so I know I missed things. It's also very easy to make technically-valid MusicXML files that contain pure nonsense (say, a perfectly overlapping crescendo and diminuendo). So in this case the goal isn't 100% coverage of every possibility, but something closer to "Synthesia should be able to play all reasonable, correctly encoded MusicXML songs". And even that will narrow further when it comes time for the sheet music to reflect the contents of the MusicXML file: Synthesia is only targeting piano sheet music.

jimhenry · Post by **jimhenry** » 09-26-21 12:59 am

when it comes time for the sheet music to reflect the contents of the MusicXML file: Synthesia is only targeting piano sheet music

Initially or forever?

For my purposes I can live with Synthesia only doing a sheet music display for the piano part but I need to have it play accompaniment parts.

Post by **Nicholas** » 09-26-21 9:15 am

jimhenry wrote: ↑09-26-21 12:59 am... but I need to have it play accompaniment parts.

That part should already work (including percussion). The "does it sound correct?" call to action includes pieces with accompaniment.

JimNYC · Post by **JimNYC** » 10-02-21 2:56 pm

Attached are .mid and .mxl versions of the same song. Here's what I noticed with the mxl file:

- the tie notes (measures 2, 6 and throughout) are playing as separate notes

- first time, it did not automatically detect left hand and right hand from the parts

- going into hands, colors and instruments, it shows 3 parts. The second part only lists 4 notes under channel 2. I had to manually assign left and right hand to the other 2.

- pickup measure did not get handled properly (you already noted this)

JimNYC · Post by **JimNYC** » 10-02-21 3:23 pm

For this mxl file, it's showing 4 parts.

Post by **Nicholas** » 10-04-21 11:37 am

Thanks for the feedback! As far as I can tell, you're the first person to try it in a week.

Good catch on the ties. I'll fix that.

For the multiple-parts situation, those MusicXML files are written using multiple "voices". In MuseScore, you choose voices using these little numbered buttons:

: voice 2.png (17.2 KiB) Viewed 30156 times

I wanted to err on the side of more control, so I have Synthesia break each voice into a separate part. (More accurately: Synthesia takes the Cartesian product of { MusicXML part × staff × voice × MIDI instrument/channel } and will break every variation into a separate part.)

I wonder if breaking down the voices should be an option. And if so, maybe it should be disabled by default? It's strange that The Little Hunters Song only has those two notes that use the separate voice. That makes me worry that voices are used more as a workaround than the way they're supposed to be used musically and that the option should be off by default.

JimNYC · Post by **JimNYC** » 10-04-21 12:04 pm

I had no idea on the voices -- those were mxl files I pulled off of musescore.com.

The staffs can easily be imploded in MuseScore, so I guess there is a balance between proper file preparation and usability. To the extent ignoring voices (I guess within the same staff) can be made an option that users can change, then it will allow users to choose how they want the file to be broken down into parts.

jimhenry · Post by **jimhenry** » 10-04-21 2:35 pm

Nicholas wrote: ↑10-04-21 11:37 am That makes me worry that voices are used more as a workaround than the way they're supposed to be used musically and that the option should be off by default.

If I remember correctly, you have to use two or more voices in MuseScore if you have different rhythms on the same staff.

Smiley · Post by **Smiley** » 10-04-21 4:39 pm

Nice to see this starting to be implemented...

I've already uploaded 3 files with midi, xml, pdf for testing ages ago...

viewtopic.php?f=15&t=10293
viewtopic.php?f=15&t=10296
viewtopic.php?f=3&t=10292

Pick-up measure seemed to work fine in my example, quaver groups did not - 12/8 not correctly grouped in three voice stems not pointing in the correct directions. You can see in the pdfs how the music should look.

Keep up the fine work ...

It's much appreciated !!!

JimNYC · Post by **JimNYC** » 10-04-21 7:27 pm

jimhenry wrote: ↑10-04-21 2:35 pm If I remember correctly, you have to use two or more voices in MuseScore if you have different rhythms on the same staff.

I don't use MuseScore all that much, but it looks to be the case: https://musescore.org/en/handbook/3/voices

Based on that (see 4th image), a piano piece would have to use multiple voices for notes played simultaneously that are different durations. So there's no way to avoid it in the mxl file generated. But it clearly is meant to just be played by a single hand. So you would definitely want it just to be a single part for the purposes of playing it in Synthesia.

Post by **Nicholas** » 10-08-21 10:35 am

jimhenry wrote: ↑10-04-21 2:35 pmIf I remember correctly, you have to use two or more voices in MuseScore if you have different rhythms on the same staff.

It's starting to sound like this has less reason to be a first-class, in-app option and more of a "don't ever split voices into separate parts unless you have a really good reason"... which is to say I'm leaving it an advanced option for the configuration window that is disabled by default. Done and done.

Smiley wrote: ↑10-04-21 4:39 pmPick-up measure seemed to work fine in my example, quaver groups did not - 12/8 not correctly grouped in three voice stems not pointing in the correct directions. You can see in the pdfs how the music should look.

Thanks for the reminder about those pieces. They just helped me find three more bugs in my MusicXML loading code.

Regarding quaver grouping and stem directions: those are sheet music things. For this preview, all we're hoping for is being able to close your eyes and still hear something that sounds correct when you load a MusicXML file. (And, well, making sure the notes fall inside the correct measures; after I "added" pick-up measures it broke Good Morning to All! I just fixed it again and now your arrangement is showing correctly.) The rest of the "does the sheet music look correct?" part is coming later.

A curious side-note: the Good Morning to All .mxl file contains two nearly-identical .musicxml files. They appear to only differ slightly in layout. And one of them contains a weird "<?Finale link="p1.musicxml"?>" tag that points to the other file. I'm not sure what's going on there, but Finale seems to be working extra hard on something. hehe.

SYNTHESIA 10.8.5667
Download from the pink box above.

CHANGES SINCE r5658

Fixed MusicXML pick-up measures.
Fixed MusicXML tied notes.
MusicXML voices are no longer split into separate parts by default. Change "MusicXML.SplitVoices" if you want it.
Older (and newer?!) Android devices should be able to use unlock keys again, despite Let's Encrypt's root certificate expiration.
The new iPad mini screen size should be recognized because I... recompiled the project using the latest version of Xcode. *sigh*

NOTE: If you'd made any hand assignments or finger hints in a MusicXML file in r5658, you may need to re-do them starting in r5667. One of the bugs had the note ordering a little scrambled, so things might not match up between versions.

This is getting close to 10.8 official. Those last two bullet points count as high-priority fixes. Google is imposing more work on me in November that I don't want to do before 10.8 goes out the door, so I pushed several under-the-hood things until next time. I've got a little bit more tidying up and the feedback you guys are giving here is excellent, so here is another preview in the meantime!

JimNYC · Post by **JimNYC** » 10-08-21 2:53 pm

Thanks for the update. I am still hearing/seeing issues with the The Little Hunter's Song's MusicXML parsing, specifically:

1) it's making the pickup meausre 3 beats (of 6/8) instead of what should be only the single 1/8th note
2) there is a pause when it does the first repeat (as it's showing 3 beats of rests in the sheet music generated), probably a result of #1
3) the ties are displaying in the sheet music as dotted half notes. The original tie is a dotted quarter with a quarter. It is then showing a dotted rest after it in the next measure.

Part of these may be that the measures are all off, and then it's trying to shove the ties into the remaining beats of the measure.

I also noticed that the tied notes are showing as dotted half notes in the MIDI generated sheet music. Probably also an issue of dealing with 5 beats of a 6/8 measure when it probably doesn't know to show them as dotted quater-quarter ties.

Attached is a PDF of the sheet music to make it easier to follow along.

Post by **Nicholas** » 10-09-21 10:05 am

1) I have an old function called BeatsPerMeasure that is supposed to take compound measures into account... for metronome ticks! I was using it to calculate the actual length of the measure. Those are different when you're in compound time. So now pick-up measures are Actually Fixed™ this time.

(Instead of seeing a 3 beat measure, it should have been all 6 with the eighth note in the last beat.)

2) This... is scary. The way Synthesia handles pickup measures is by "filling up" the first measure with silence until the pickup notes. Because it can't imagine a measure that isn't its full length according to the time-signature. It's not particularly graceful, but it's done the job for 15 years. That repeat in The Little Hunter's Song means it finally doesn't work anymore.

I'll have to think about how to tackle that. This isn't something I can just brush off because after the repeat nothing fits to the measures anymore. It might be time to teach Synthesia about partial measures. Hmm.

3 and later) This is sheet music stuff again that you should be ignoring!

Synthesia has never known how to draw tied notes. It will learn soon. But not in 10.8.

(I had gotten through the fix for #1 and nearly clicked my "make another dev preview!" button before I saw #2 and now I'm worried about whether I'll be able to make my November target.)

In the meantime, again, thanks for the feedback! More eyes on new work is always better!

JimNYC · Post by **JimNYC** » 10-09-21 10:47 am

Can't the pickup measure beat(s) just be tacked on to the end of the last measure of the repeat? That's all it basically needs to do as that one is 5 beats and then it just needs the 1 beat pickup to make it a complete measure before it does rest of the repeat full measures all over again. So in terms of playback, you really only have the first pickup measure followed by full measures until the end of the second repeat.

Alternatively, the midi playback seems to handle it correctly without the pauses. Looking at the midi sheet music at least, it seems to add in time signature changes of 1/8 and 5/8 to handle the pickup and last measure. It plays back correctly, so couldn't the MusicXML just parse it similarly to end up with something similar?

Hopefully either option isn't too scary.

Post by **Nicholas** » 10-09-21 2:05 pm

Inserting faked/incorrect time signatures has been a workaround in the past. (A few of the built-in songs have a 1/4 or 1/3 measure at the very beginning.) And if this gets too hairy, it is definitely something to fall back on.

Although, I like the idea of loosening my infrastructure to the point of handling arbitrary measure sizes. A recurring theme I've discovered in this MusicXML work has been that I get to throw off all of these rigid algorithms I've had to build up around (poorly specified) MIDI files and instead I can now just trust what I find in the file. If they say there's a new measure after one beat, who am I to argue?

The idea is that MIDI doesn't have any concept of a "measure"! Note events just sort of start happening and it's up to Synthesia to decide where the measure lines should go. (So there's no such thing as a pick-up measure. If you don't "fix" it by inserting leading silence or a fake 1/x time signature, there is no way to know!) But MusicXML has them perfectly explicitly defined, so there is zero guesswork.

The best part is that in the code, there is a very pronounced line between "figuring out where the measures should be" and the rest of everything that relies on the measure list, how it relates to time (both musical "pulses" and wall-clock microseconds) and when things need to change. All I need to do is make that first half conditional on whether Synthesia is loading a MIDI or not. If it's not, it can just use the measure list from the MusicXML file verbatim. (The scariest part now is that each part/staff technically defines its own measure list. The best I can do is cross my fingers and hope that while parsing each staff, the same list of measures were found in each one!)

jimhenry · Post by **jimhenry** » 10-09-21 5:55 pm

Regarding repeats that start with pickup notes, the formal name for which is anacrusis:

I was surprised that repeats don't have to coincide with bar lines. But based on the Little Hunter's Song, it seems like the repeats have to be placed so that they maintain the length of a full measure. In Little Hunter's Song, since the implicit start of the repeat is the pickup beat 6, the end of the repeat is placed after beat 5. In this schematic of the beats, bar lines, and repeats, the repeats are underlined:

|: 6 | 1 2 3 4 5 6 | ... | 1 2 3 4 5 :| 6 | 1 2 3 4 5 6 | ... | 1 2 3 4 5 |: 6 | 1 2 3 4 5 6 | ... | 1 2 3 4 5 :| 6 | 1 2 3 4 5 6 | ... | 1 2 3 4 5 ||

From Wikipedia: Western standards for musical notation often include the recommendation that when a piece of music begins with an anacrusis, the notation should omit a corresponding number of beats from the final bar of the piece, or the final bar before a repeat sign, in order to keep the length of the entire piece at a whole number of bars.

Extrapolating from the Wikipedia explanation, I think two forces are at work. The barlines marking off full measures that define where the strong and weak beats fall and the musical phrasing. My extrapolation is that musical phrases have to be a whole number of bars but they don't have to start with a strong beat. When a musical phrase does not start on the strong beat of beat 1, you use pickup notes to shift the musical phrasing relative to the musical beats. The shifted phrase will end before the end of a full measure, hence the last measure is shortened to match the phrasing, which also corresponds to the amount added as pickup notes.

Hopefully this understanding of phrasing that is not aligned with measures will lead to an implementation that is elegant and robust.

Post by **Nicholas** » 10-12-21 2:22 pm

jimhenry wrote: ↑10-09-21 5:55 pmHopefully this understanding of phrasing that is not aligned with measures will lead to an implementation that is elegant and robust.

Even taking into account a few other peculiarities I just discovered (it's apparently typical not to "fast-forward" the MusicXML "cursor" to the end of each measure after a "backup" to fill in other voices--which was leading to a different set of detected measure times between each staff!), the new code that reads arbitrary length measures is shorter and simpler than the pick-up measure specific hack I had before. So everything worked out in the end.

: A trimmed down snippet of Little Hunter's Song (with just the pick-up, repeats, and ending intact), screenshot from MuseScore; musescore.png (12.37 KiB) Viewed 29885 times

: "Export as MIDI" shown in Synthesia; midi.png (18.33 KiB) Viewed 29885 times

: "Export as MusicXML" shown in Synthesia; musicxml.png (13.83 KiB) Viewed 29885 times

The lack of faked time signatures makes the MusicXML export much less noisy!

jimhenry · Post by **jimhenry** » 10-12-21 3:22 pm

The additional barline in front of the pick-up note is jarring. How hard is it to not have that?

Post by **Nicholas** » 10-12-21 3:38 pm

It won't be jarring, eventually. The repeats will show correctly so the 1/8 and 5/8 measures will appear exactly like the original sheet music. So that's only a temporary inconvenience.

(And in the meantime, the answer is "very hard".)

JimNYC · Post by **JimNYC** » 10-12-21 7:22 pm

Nicholas wrote: ↑10-12-21 2:22 pm So everything worked out in the end.

That's good to hear.