Page 11 of 11
Posted: 09-09-21 10:18 am
by jimhenry
Nicholas wrote: 09-08-21 3:03 pm Crescendo/decrescendo don't have any explicit before/after dynamics specified in MusicXML. Maybe I'm just not clear on the rule. Does a crescendo step the current dynamic up one "step"? Say, from mf to f? Or from pp to p? Is it two steps? What does the dynamic increase to if a crescendo follows an fff? Do any of those answers change if it's a "long" (say, multi-measure or even multi-line) direction vs. a "short" one?
Based on a quick look at the results of a Google search for "How do you notate a crescendo?", my previous short look at MusicXML in connection with pedaling, and hazy recollection from my music arranging course, this is my best guess on how you handle crescendo and decrescendo.

From https://www.masterclass.com/articles/ho ... o-in-music:
Crescendos can appear in sheet music notation in two forms—one precise, the other leaving room for a player's or conductor's interpretation.

Composers indicate the looser form of crescendo notation with the actual word crescendo (abbreviated cresc. on musical scores). The word appears at the point where the music begins to increase in volume. Ideally, the musical score will also indicate the climactic point of the crescendo passage by showing the dynamic marking the crescendo is building up to, such as forte (f) or mezzo-forte (mf). However, be aware that not all musical passages contain such dynamic markings, which leaves the crescendo marking more open to interpretation.

The more precise form of crescendo notation involves symbols that look like hairpins, inequality symbols, or alligator jaws. In most cases, "hairpin" is the standard term musicians use. Hairpin crescendo symbols show the crescendo’s exact length and the prescribed volume at the beginning and end of the crescendo.
Based on my superficial understanding of MusicXML, there would be from 1 to 3 elements marking a crescendo.
  • The starting element, the word or the starting location of the hairpin
  • The ending element, the penultimate dynamic if started by a word, or the ending location of the hairpin
  • The penultimate dynamic if using a hairpin
The penultimate dynamic is optional, leaving room for a player's interpretation.

I assume the issue is performance of a MusicXML crescendo that lacks a penultimate dynamic. The worst is crescendo notation with the word and no penultimate dynamic because there is no indication of where or at what level the dynamic change ends. I don't think anyone expects much from a Synthesia interpretation of such notation.

Where the length is not specified, one measure is probably an acceptable default.

Where the penultimate dynamic is not specified, a change of two dynamics, e.g. p to f, might be a good guess. BUT the lack of a penultimate dynamic means no dynamic marking anywhere following the crescendo marking. Otherwise, that dynamic is presumptively the penultimate dynamic.

This table, from https://en.wikipedia.org/wiki/Dynamics_(music), relating dynamics to MIDI key velocities might be useful:

Code: Select all

Symbols	                        pppp	ppp	pp	p	mp	mf	f	ff	fff	ffff
Logic Pro 9 dynamics[24]		16	32	48	64	80	96	112	127	
Sibelius 5 dynamics[25]		        20	39	61	71	84	98	113	127	
MuseScore 3.0 dynamics[26]		16	33	49	64	80	96	112	126	
MakeMusic Finale dynamics[27]	10	23	36	49	62	75	88	101	114	127
SmartScore X2 dynamics[28]	29	38	46	55	63	72	80	89	97	106
SmartScore 64 dynamics[29]	30	40	50	60	70	80	90	100	110	120

Posted: 09-10-21 12:50 am
by Bavi_H
jimhenry wrote: 09-09-21 10:18 amthe penultimate dynamic
(Note "penultimate" means "next to last". Perhaps it might be clearer to just say "final" or "ending" instead?)

Posted: 09-10-21 3:09 pm
by Nicholas
jimhenry wrote: 09-09-21 10:18 amThe worst is crescendo notation with the word and no penultimate dynamic because there is no indication of where or at what level the dynamic change ends. I don't think anyone expects much from a Synthesia interpretation of such notation.
I agree. Most of the "it's just written as an abbreviation above the music" stuff is very free-form in MusicXML. Most things have a specially-named XML element with attributes that contain everything that's necessary in nice, machine-readable formats. But "cresc.", "poco rit.", "Ped. simile" and many others are just... text. (This is probably why other apps have gone down the path of using a sidecar MIDI file where the sounding part can probably be expressed a little more precisely.)

jimhenry wrote: 09-09-21 10:18 amWhere the length is not specified, one measure is probably an acceptable default.
That's more generous than my plan. I was just going to ignore it if the length wasn't specified! :lol: Now I'm going back and forth. I suppose the result of following my plan is always incorrect. Your suggestion of defaulting to one measure might sometimes be correct. I suppose that's better.
jimhenry wrote: 09-09-21 10:18 am... BUT the lack of a final dynamic means no dynamic marking anywhere following the crescendo marking. Otherwise, that dynamic is presumptively the final dynamic.
Doing a kind of "scan ahead" looking for dynamics can get tricky. Where is the cutoff? A dynamic belonging to the same measure that the [de]crescendo ends? One measure past that?
jimhenry wrote: 09-09-21 10:18 amThis table [...] relating dynamics to MIDI key velocities might be useful
Thank you. I remember spotting that a month or so ago, but had lost track of the link. What's a little scary is how much variation there is in the table! I would argue there are some obviously "wrong" choices in some of those lines, but nobody asked me. :lol:

(If four f's isn't velocity=127, then I don't know what is, SmartScore X2!)

dynamics.png
dynamics.png (19.1 KiB) Viewed 101829 times

Posted: 09-10-21 6:29 pm
by jimhenry
Bavi_H wrote: 09-10-21 12:50 am
jimhenry wrote: 09-09-21 10:18 amthe penultimate dynamic
(Note "penultimate" means "next to last". Perhaps it might be clearer to just say "final" or "ending" instead?)
Serves me right picking that fancy word without really knowing the exact meaning. :oops:

Posted: 09-10-21 7:09 pm
by jimhenry
Nicholas wrote: 09-10-21 3:09 pm
jimhenry wrote: 09-09-21 10:18 am... BUT the lack of a final dynamic means no dynamic marking anywhere following the crescendo marking. Otherwise, that dynamic is presumptively the final dynamic.
Doing a kind of "scan ahead" looking for dynamics can get tricky. Where is the cutoff? A dynamic belonging to the same measure that the [de]crescendo ends? One measure past that?
I think you are envisioning a case I forgot about, a hairpin with no dynamic marking at the end of the hairpin. I thing this is a case where you know the start and the end locations but not the end dynamic. While I think it is true that a hairpin is sometimes used more as an alternative to the word and only indicates the start location, I think it is acceptable to say that a hairpin will be performed by Synthesia exactly at the length given. If there is no dynamic at the end, then the two step change assumption will apply.

Treating word only crescendo markings as 1 measure crescendos of 2 steps might not be the too bad. If you are a little more ambitious, scanning 4 measures ahead would probably catch most ending dynamics that end a crescendo.

Posted: 09-13-21 2:37 pm
by Nicholas
jimhenry wrote: 09-10-21 7:09 pmI think you are envisioning a case I forgot about, a hairpin with no dynamic marking at the end of the hairpin.
This seems to be the most prevalent case across the various files I've been able to dredge up. :? You're right that at least the duration is well-specified in that case.

The rest of your advice also sounds good. I haven't had to do any sort of "scan ahead" yet, but it shouldn't be too awful. A quick perusal of how these editors seem to encode things even when the dynamics are placed directly in front or behind the hairpin (as part of the same element or later "snapped" to it) shows that there is very little consistency in how that will make it eventually show up in the file. So being able to handle everything in a more general way is probably for the best.



In the meantime, a new anecdote while loading various "real world" MusicXML I've been able to find: I guess some repeat dots are implied?

Schumann_The_Wild_Horseman_Op._68_No._2.pdf
(44.01 KiB) Downloaded 2404 times
Schumann_The_Wild_Horseman_Op._68_No._2.mxl
(4.17 KiB) Downloaded 2357 times

I loaded that and noticed Synthesia only showed the first 8 bars. Without the leading repeating dots on measure one, it didn't realize it was already inside a repeat, so when it reached the repeat-end, it just kind of ignored it and moved on. Immediately after there are begin repeat dots, so my strategy is to begin storing everything that is going to need to be repeated in a separate buffer off to the side, so that it can be "stamped out" as many times as needed later... but then there is never an end-repeat again, so no copies are ever stamped out into the "real" song again. Then the song ends, the off-to-the-side repeat buffer is discarded and all that's left are the first eight (not-repeated) bars! :lol:

This sounds like a job for more scanning ahead...

Posted: 09-14-21 9:26 pm
by jimhenry
Nicholas wrote: 09-13-21 2:37 pmI loaded [the Soldier's March] and noticed Synthesia only showed the first 8 bars. Without the leading repeating dots on measure one, it didn't realize it was already inside a repeat, so when it reached the repeat-end, it just kind of ignored it and moved on.
[Having reread your post after writing and posting my reply, I think I understand your post better. I have amended my reply accordingly but I didn't completely revise it based on my new understanding. So that's my excuse if this reply seems a bit confused.]

Did you notice that both the PDF and the mxl file appear to have a typo? The final bar line at the end of measure 24 is supposed to be a repeat bar line according to every edition I could quickly find on-line. I don't think a beginning repeat bar line without a matching ending repeat bar line is even a proper syntax.

I think I understand you to say that Synthesia was not prepared for the case of an ending repeat bar line (at the end of measure 8) without a beginning repeat bar line, which means repeat from the beginning. This is pretty common and it will be important that Synthesia handles it correctly. I think your side buffer strategy has to always start by saving from measure 1. If it encounters a beginning repeat bar line, then the buffer is cleared and saving restarts from the just found beginning repeat bar line.

I now understand you to say that Synthesia starts collecting notes when it encounters a beginning repeat bar line but does not display those notes until it finds the matching ending repeat bar line. This resulted in a display of measures 1-8 only. Arguably omitting a display of measures 9-24 is not unjustified since a beginning repeat bar line without a matching ending repeat bar line is incorrect notation. But it would probably be better to display the measures once when confronted with such notational errors.

I can think of two approaches.
  • The first is to display everything as you encounter it for the first time and only rely on the buffer for second and subsequent displays. I think that corresponds to the way a musician would sight read, play everything as you go and then use the repeat markings to circle back and do it again.
  • The second approach is the counterpart of the handling of an omitted beginning repeat bar line at the start of measure 1. Where you have an explicit beginning repeat bar line, treat the final bar line as an ending repeat bar line if there is no explicit ending repeat bar line. While I dislike the second approach because it seems to only be there to handle incorrect notation, maybe it is necessary if everything is initially buffered because of the possibility of an omitted beginning repeat bar line at the start of measure 1. When there are no repeat bar lines at all, the entire song is output from the buffer when the the final bar line is reached.
You didn't mention multiple endings. I am pretty sure those will create some issues. It is not unheard of to have two endings with more than two repeats that use the endings in various orders.

Nested repeat bar lines are permissible but they are held in low regard by most musicians and computer programs that have to deal with them. I wouldn't fault you for just saying Synthesia does not handle them correctly and accepting whatever result is produced by nested repeat bar lines. Of course, if you are a LISP wizard and you enjoy recursive solutions, then knock yourself out correctly handling nested repeats. Here's one to practice on: Image

Posted: 09-14-21 10:22 pm
by Nicholas
jimhenry wrote: 09-14-21 9:26 pmI think your side buffer strategy has to always start by saving from measure 1.
This one isn't so bad to handle as a one-off situation. Because it's XML and the parser I'm using already supports XPath, it's going to be a one-liner to say "find the first <repeat> element in the document, if any" and one more line to check if it's an ending. If so, I can initially point the parser at the repeat buffer instead of the real song.
jimhenry wrote: 09-14-21 9:26 pmNested repeat bar lines are permissible but they are held in low regard by most musicians and computer programs that have to deal with them.
Yeah, that image is a mess! I agree with you that Synthesia shouldn't support it. :lol: (My whole "repeat buffer" strategy would need to change to do any sort of nesting... and it's lovely just as it is. hehe.)

As for the missing repeat dots at the end, I still need to change something to not "lose" the measures. Maybe if there is no repeat at the end, the side buffer just gets inserted once instead of twice? Then it's not Synthesia's bug anymore, at least. If they really intended a repeat, they can modify the file!

EDIT: MuseScore agrees with me. Their "unroll repeats" feature also just keeps the final section once and effectively ignores the "unterminated" repeat.

EDIT 2: They were one-liners as expected and both species of strange repeat (one typical, one malformed) are well-behaved now.

EDIT 3: I just noticed you were revising your post while I was working on this one. It looks like we came to similar conclusions. (Regarding multiple endings, I omitted the details but there are additional buffers for each ending chunk and a neat facility for tacking these things together like chain links to handle all of it rather gracefully. I first mentioned it about a page or so ago. All of that already works as-expected and was probably the highlight of this whole initial effort.)

Posted: 09-16-21 3:24 pm
by JimNYC
Attached are the PDF and MXL files I had for Solider's March. Yes, there is a end repeat at the end.

In any case, glad to hear progress is being made. Will Synthesia display finger notation in the sheet music when importing from MusicXML?

Posted: 09-16-21 3:31 pm
by Nicholas
JimNYC wrote: 09-16-21 3:24 pmWill Synthesia display finger notation in the sheet music when importing from MusicXML?
Eventually, yes. Not in this first preview though: this first pass is a quick "load MusicXML files essentially as MIDI files" with none of the additional information used in the sheet music display (yet).

This has been an excellent exercise though. I went from knowing only the top-level ideas about the format to many of the nitty-gritty details. I'm glad I decided to follow jimhenry's suggestion of doing this half sooner because it has already completely transformed the way I'm planning to bring everything over to the sheet music display (in a good way).

Posted: 09-16-21 4:46 pm
by JimNYC
OK, I guess I got confused about the intended workflow that you had in the other MusicXML thread:
Today
MIDI → [internal] → glyphs

Possibility A
MIDI → [expanded internal] → glyphs
MusicXML → [expanded internal] → glyphs

Possibility B
MusicXML → glyphs
MIDI → [internal] → simple, interpreted MusicXML → glyphs

Since you are supporting stand-alone MusicXML files (as opposed to the hybrid approach of MIDI + MusicXML that others have adopted), I guess you needed to a way to parse MusicXML to MIDI for the sound part.

Posted: 09-16-21 7:56 pm
by Nicholas
You are correct. Having a sidecar MIDI file still creeps me out because it feels like it would be more work matching the notes between the two, especially with MusicXML's time "cursor" constantly moving forward and backward each measure for each different voice and part. And there are things with ambiguous timing like grace notes that could throw more wrenches in the gears.

In terms of convenience, if we required anyone with just a MusicXML file to "just open it in your favorite editor and export it as MIDI so Synthesia can load both", not only have we added an extra step for our users, an extra file to tend, and a dependency on a third-party app (which gets complicated fast for tablet-only users without a desktop computer available), but that MIDI file will never contain anything the MusicXML file doesn't anyway. Worse, I've discovered that some apps (like MuseScore) omit certain details (like ignoring [de]crescendos) in their MIDI exporter. So Synthesia can do a better job on its own with just the MusicXML, anyway. Fewer steps, fewer dependencies, fewer files, (potentially) better quality output.

What changed in the meantime with my new experience is that Possibility B is the (very) obvious choice now. The MusicXML schema maps almost directly to Synthesia's existing glyph infrastructure. So the file is going to come in with the notation basically already assembled. So all of the existing MIDI-to-sheet-music heuristics will still be used exclusively for MIDI, the way you indicated.

The only real work will be pulling apart two layers that I only just recognized as being distinct: the MIDI-to-sheet-music heuristics and the sheet-music-layout code. There is the matter of spacing, justification, and pagination that has to be re-run each time you adjust the sheet music zoom. That is always necessary, no matter the source of the glyphs. That distinction between generating the glyphs and laying them out has been a little fuzzy in the past. I just need to draw a clearer line between the two and it should be ready to go.

There are still some additional technology pieces to add: Synthesia can't draw slurs/ties. And the reason it only ever beams two notes at a time is because I still have yet to teach it how to shorten/lengthen the stems of the intermediate notes to meet the beam. Behind Bars has several pages just discussing how to choose the angle of the beam! That falls under layout, too, because besides "the stem is upright" and "a beam exists between these four notes", there isn't any information in MusicXML about the exact placement of things like beams or slurs.

Posted: 09-16-21 8:51 pm
by JimNYC
I get it, I guess I was just thinking of the MusicXML file as the sidecar file initially to achieve better sheet music display, rather than the other way around since the MIDI part already works for playback. So it was more like if you want better sheet music, then provide a matching MusicXML file which tells it how to display. With that approach, the playback and display can go out of synch if they aren't generated from the same source. It's sort of like a chicken and egg paradigm, and if you are going to support standalone MIDI and MusicXML anyway, then each has to handle both playback and display ultimately regardless.

However you get there, I think many of us will be ecstatic as long as you feel improved sheet music notation display can be done!

Posted: 09-17-21 12:18 am
by Nicholas
That viewpoint--which I don't think I've ever considered in ~10 years of talking about MusicXML--does make a lot of sense. My inability to see it is probably rooted in the same fear-of-mismatch between the two files: in my head the two displays must always be in absolute lock-step. The idea is that you should be able to turn off either the falling notes or the sheet music and still be able to play unimpeded. Allowing the sheet music to "go out of sync" kind of relegates it to second-class status, which is something I'd like to avoid.

I'll be curious to see how the no-sidecar decision plays out in the long run. Both file formats are very capable. The venn diagram of what they can and can't express (from a falling note block perspective) is almost fully overlapping. All MusicXML misses out on is a bit of nuance in the performance and being able to generate some of the more esoteric MIDI controller messages. It's a trade-off that--in the extreme case that it turns out too great--could always be corrected at some point in the future.

In the meantime, Synthesia will still be able to load either kind of file. So if you're looking for an expressive performance, load a really high-quality MIDI file. If you want beautiful sheet music, load a MusicXML file. :)

Posted: 09-17-21 10:16 am
by JimNYC
Nicholas wrote: 09-17-21 12:18 am In the meantime, Synthesia will still be able to load either kind of file. So if you're looking for an expressive performance, load a really high-quality MIDI file. If you want beautiful sheet music, load a MusicXML file.
That's great. My only concern is one of timing then...I hope that while there is momentum with MusicXML that some of the actual sheet music display updates make their way to the preview builds. Thanks for continuing to improve this!

Posted: 04-21-22 1:10 pm
by Nicholas
I saw an interesting bug report today that makes this MusicXML file crash Synthesia.

That particular song has a bit of time-travel involved. If you look at measure 61, you'll see two tied Cb notes in the bass staff. Inside the file, they're given backwards: the second note is described first with its "this is the end of a tie" indicator. That is followed by a "rewind a little bit" instruction and then the first Cb with its "this is the start of a new tie" indicator.

The crash is because Synthesia expects ties to be started first and then stopped later. So when it saw the "stop this tie" when it didn't know about any Cb tie yet, that caused the confusion.

So I'm going to have to add something that can retroactively reconnect closed ties from the future. Or something. :anxious:

Posted: 11-26-22 1:05 pm
by Electrode
How about when Synthesia encounters a tied note (whether it is the end or beginning of a tie), it does a quick check either side of that note to find out where the other corresponding note of the tie is (e.g. checking for the same pitch with a corresponding "tie" instruction), then reverses the order of the instructions if necessary and processes as normal?

Posted: 11-28-22 4:29 pm
by Nicholas
Electrode wrote: 11-26-22 1:05 pm... a quick check either side of that note...
Because of the way the <backup> and <forward> elements can work (arbitrarily skipping around in a measure), it's not uncommon to see an entire voice played through normally to the end of the measure before backing up to the start to do the next voice, etc. So, it's not hard to imagine a scenario where the "quick check" might have to sift through (up to) hundreds of MusicXML elements to be sure. :grimace:

I actually ended up with a nice solution to the problem where I just maintain a list of closed-but-never-opened-in-the-first-place ties while reading through the song. Then, whenever Synthesia encounters the start of a tie, it checks for a matching "future tie" to connect it to first. That's been able to handle all the situations I've encountered so far.

Another elusive, tie-based, but unrelated bug (that has existed since 10.8 first added MusicXML loading) that I've been chasing down for over a week now has to do with ties across repeat endings:

tied repeats.png
tied repeats.png (24.47 KiB) Viewed 89665 times

I was happy to see that Synthesia 10.8 was already handling that situation correctly in most situations. It's only in a particularly convoluted case where there are repeats basically everywhere, butted up against each other, all with multiple endings where one tie deep in the middle isn't reconnected correctly. Whenever I try to pare down the problem to a simpler case, it suddenly starts working and the crash goes away. So it's been a bit of a slog trying to figure out exactly where the corner case is. (This--and my whole family catching the flu two weeks ago--is actually the reason 10.9 hasn't gone out the door yet.) :?

Posted: 12-21-22 2:56 pm
by Nicholas
Wow, between our kids and me/Lisa being alternately sick almost constantly for the past month and the problem being super hard to track down, I can't believe it took that long to find the ties-across-repeats bug. But I just fixed it. Well... more like "fixed" it. :anxious:

If not for the "ties from the future" quirk, it would have been easier, but I don't think the behavior is unreasonable now. There's still one situation where an incoming tie at the start of a repeat that doesn't have a match at the end of a repeat might result in the orphaned note (the end of a tie that doesn't have a beginning the second time around) simply being dropped instead of being shortened the second time around.

... but the app doesn't crash anymore. And that was my hard requirement for releasing 10.9. So now I can release 10.9. Release imminent. Moments before the busiest Synthesia day of the year. I'm sure this won't cause any problems. :lol: