Synthesia 11

Try new versions before anyone else!
Always the latest dev version: WindowsmacOSiPadAndroid (or .apk)
Please report comments and bugs!

Your data hasn't disappeared: PC/Mac development previews store their data in a different place. Details here.
Posts: 13166

Post by Nicholas »

Not yet. Synthesia 10.5 was just last month and there have been seven bug-fix updates to the Android version since then! (It's been a busy holiday season so far!)

The good news is that the very next Synthesia 11 dev preview is the one that will allow full-screen sheet music, so there shouldn't be too much longer of a wait.
Posts: 15

Post by spstarr »

Looking forward to trying this out Nicholas, very much so, great work :D
Posts: 64

Post by Korados »

Nicholas wrote: If you're still getting clipping, you might even try reducing the volume on the Settings --> Built-in MIDI Synthesizer screen down below 100%. That may solve it.

So, a bit of nomenclature: both Synthesia and VirtualMIDISynth use BASSMIDI under the hood. In this case, "I guess Bassmidi doesn't preload..." is contradictory because "... like Coolsoft's Midi Synth" implies it does. ;)

What you mean to say is that Synthesia doesn't preload the fonts. And you would be correct. Until the next silent 10.5 update where it will preload fonts by default (but let you turn it off if you don't want to multiply Synthesia's RAM use by 10-200x when loading giant SoundFonts). :D
Oh, I wasn't aware of that. I only use the General User Soudfont, and it's only 35 MB big, so I guess preloading it wouldn't use so much RAM. I'm looking forward to the next update!
I wish you a happy new year!
Posts: 135

Post by revilo2 »

In the final version of Synthesia 11, the display of the score will be the same as a musescore's score ?

What about barlines ?
Posts: 13166

Post by Nicholas »

The goal is to be very close to MuseScore's display. It won't be 100% identical, but I'll take 95% instead of what is probably closer to 20% today.

What did you mean about bar lines? How are they different?
Posts: 135

Post by revilo2 »

I'll will try to explain this in english (not easy for me) :

I was talking about the barlines of the score : "Start repeat" and "End repeat"

Musescore display the whole score and manage the barlines as a musician will do :

When the cursor is at the "End repeat" position (for the first time), it return the the 'Start repeat" position.

But Synthesia doesn't display the whole score.... :)
Posts: 13166

Post by Nicholas »

Repeated sections are on the list for Synthesia 11 with a MusicXML file that contains them.

For MIDI files it would be trickier because we'd need some algorithm for comparing measures and detecting repeats manually (because that information isn't stored in a MIDI file), so I'm holding off on those for now.
Posts: 15

Post by spstarr »

For the Sheet music mode, will we be able to show the letters of the notes on the staff? This looks great so far!
Posts: 13166

Post by Nicholas »

The musical font we're using includes note heads with labels, so it's a possibility. (You'd have to scale the music up pretty far to make any use out of them though.)

So far I've only got a note on my internal Synthesia 11 roadmap that they exist. It's more of a "stretch goal" at this point with how long Synthesia 11 development has already been ongoing and how much is still left. If it doesn't make it into 11.0, I'm hoping to clean up any of the missing features during a few small 11.x updates that will follow shortly afterward.
Posts: 135

Post by revilo2 »

and fingering ?
Posts: 13166

Post by Nicholas »

That one will definitely be included. Those are just little numbers above/below the staff, and Synthesia is going to have to know how to draw a variety of text styles there already for other things, too.
Posts: 13166

Post by Nicholas »

Progress Update!

I haven't checked-in in a while. Here's what's been going on during April and May 2019:


Ever since 8.2 back in 2012 when Synthesia adopted most of iOS's menu conventions, I've been using the little check mark at the right side of lists both correctly and incorrectly. When you're choosing one of several options, the checkmark is the right answer:
goodCheckmarks.png (4.89 KiB) Viewed 104147 times
But when a check mark just means "on" or "off", that's confusing:
badCheckmarks.png (8.96 KiB) Viewed 104147 times
It's not clear at all that you can toggle those independently. Instead, confusingly, it looks like you're choosing between the two.

Having a proper "switch" control was long overdue. They make the above distinction immediately clear and remove 80% of the work of changing options on the Settings screen. Virtually all of the "drill down to a sub-menu to choose between two options" can be collapsed outright into a cute little switch:
switches.png (14.31 KiB) Viewed 104147 times
One caveat: the sub-menus had a little "(Default)" to indicate a default setting. With switches, that information is lost. On the Advanced settings screen, this might cause some trouble if someone just goes through and turns them all on (while poking around or whatever) and inadvertently causes something to misbehave. So, to make sure everything is covered, there is a new "Reset to Defaults" button at the bottom of both the Advanced and Gameplay tabs that will restore those things to their default, "safe" values.

Rendering Engine Work

The switches were actually an excuse to test some new technology that is being added to the back end by our new graphics developer contractor. (For anyone that does graphics and is curious: static vertex buffers that persist between frames instead of "dynamic everything all the time!") This is the technology that will eventually solve the very last of the major inefficiencies in Synthesia. Once the falling notes are all in static buffers that don't need to be sent to the graphics card each frame, the app should realize some big improvements in battery usage and smoothing out the last of the frame rate hiccups. The minimum hardware target should drop pretty substantially, too. Something with the performance of a Raspberry Pi might not be out of the question (porting to a new platform notwithstanding).

Synthesia Support Assistant

I don't like taking time away from working on the app, but the support workload has been steadily increasing for 12 years now and it was time for something better than the little troubleshooting drop-down that used to be on the support page. The new support assistant can do a few cool things right now (including completely automatically issuing an unlock refund), but the best part is that it's super flexible and I can trivially add new answers to the little Q&A tree; much easier than the old system. The assistant is also going to learn how to do a few more neat tricks like updating email addresses for older unlocks and a few other time-saving things to hopefully reduce the email workload even further.

This last one actually took the better part of last month, but it was a lot of fun. I got to play with WebSockets for the first time and the tool has a lot of room to grow. The ask/respond format there can handle just about anything in the future and a chat window is a very familiar interface these days. A single click/tap from a row of choices is a much friendlier UX interaction (especially on a touchscreen) than messing around with drop-down boxes.

The changes there actually helped make the whole support section of the site make a lot more sense. Instead of separate FAQ, support/troubleshooting, and guide list pages where everything felt scattered to the wind... everything is in one place now: the guides are listed next to the support assistant, which in turn can answer all the old FAQ questions. For things that were only masquerading as frequently-asked but were actually high-level project information, I made a new (also long-overdue) About page.

Ancient Website Migration

Since 2014 (around the time we added the Music Store), the website has actually been split between old stuff (ASP.NET WebForms) and new stuff (ASP.NET MVC). For the last couple years, something like 95% of everything had been migrated over. In a recent attempt to finish that migration, I thought it would be safe to finally switch off the ancient version of the "unlock via short code" process that hasn't been used since Synthesia 10.2, nearly five years ago... and got to learn firsthand how many people are apparently still using those older versions by the flood of email asking why unlocking doesn't work anymore.

In any event, I just spent a couple days migrating the old to the new in a more full-featured way so Synthesia 10.2 and earlier can be unlocked via short code once again and the WebForms site can finally be shut down. One less thing to support, keep track of, and maintain. Whew.

MIDI Input Latency Investigation

Since day one, MIDI input has been handled on the UI thread. This seemed reasonable at the time when it was a hobby project by a kid just a few years out of college. It's not reasonable anymore. Including the BASS-based synth already helped things dramatically: a "default" user that just downloaded the app for Windows saw latency drop from 240ms down to 80ms. That's a world of difference, but I can squeeze a little harder.

I put a trace in the code to measure how long it took Synthesia from seeing MIDI input from the OS until it was processed through the main (graphics) thread and sent back out to the OS as an output message. At 60 frames per second (16.6ms per frame), you'd expect the average to be about 8ms... and it was. Curiously, there were two peaks in the chart:
inputLatency.png (14.56 KiB) Viewed 104146 times
That's a tiny image, but the peaks are at 5ms and around 15ms. The more intense the test, the more that ends up landing in the 15ms range. Worse, a few samples spiked out into the 30ms and 40ms range! I think my least favorite part is how wide the base of that graph is: sometimes your notes are played in 2ms. Sometimes they take 19ms. That's a lot of jitter that is jarring at best. This is one of those things where even if you didn't notice it, subconsciously you were having a poorer experience.

These are all things that can be solved by driving the MIDI listening part of Synthesia in a more event-driven way where it's handled (and sent back out) immediately after it arrives. Once fixed, the graph should just be a single tiny spike below 1ms. That will be a lot of under the hood work (that is not planned for 10.6), but with any luck, eventually we should be able to drop the "default" latency that a user sees by another ~15ms. Every little bit counts. (Best of all: if you happen to use a real digital piano instead of a software synth, Synthesia's own 15ms or so is the majority of your wait, so fixing this should make a huge difference in those cases.)

All of that is a bit of a grab bag. Here's a rough list of things that will be done before the next 10.6 preview:
  • Three new "actions" for the support assistant (again, I don't like giving up app development time, but this actually increases velocity in the long run!)
  • Sprinkle the new UI switch controls in about 8 more places.
  • ~11 more improvements/fixes suggested here or via email. Most of them are very small. A few of them are medium (and should be pretty exciting).
Posts: 13166

Post by Nicholas »

Progress Update!

Here's what's been going on from June through August, 2019:

Slowly Evolving My Development Philosophy

With the advent of the Embarrassment Driven Design post, I've been thinking about how to deliver things to you guys more regularly and more reliably. I've kept an eye on the length of my internal task list (specifically, the line just after "Synthesia 11 Complete") and despite continually completing/knocking tasks off the list, that line number has continued to grow instead of shrink. Said another way: new "important" tasks have been rolling in faster than I can complete them. :? Better triage would go a long way here, but I have an idea (that I vaguely recall jimhenry might have already suggested, going on three years ago now) that I think will help even more:

Instead of one monolithic Synthesia 11 update (that is apparently too large to ever finish), it will be more tractable to impose one major sheet music improvement on each small update. In 10.6, that is full-screen sheet music. In 10.7 that will be correct stem directions. In 10.8, well... you get the idea.

By chipping away at the sheet music task list while I go along cleaning up the other embarrassments, hopefully everything will improve more rapidly. MusicXML support is still the headline feature of Synthesia 11, but hopefully by the time we get there, the (generated from MIDI) sheet music will be in such good shape that much less work will be required.

Card Testing and Other Website Work

Some bad actor had started using the Synthesia website for card testing, which means automating the checkout step on the website to try tens of thousands of usually-bad card numbers per day, looking for "good" ones they can use somewhere else. So there was some extra work involved to detect that sort of behavior and nip it in the bud.

I taught the Support Assistant how to automatically update an email address for an unlock key (if you've got the short or long code) and to shorten the process for us dramatically if you've only got the old email address.

During that work, I improved the error tracing on the site, which exposed a few more rare errors that could happen in special circumstances. So, another week or so was spent cleaning those up.

The 10.6.5239 "Switches and Paste" Preview

But you guys already knew about that one.

More Engine Improvements

The graphics work I've been contracting out has continued to be awesome. I was able to start using another new feature (texture atlases) immediately to start reducing the amount of extraneous work on graphics hardware. For now it's a small impact, but this is one of the key features in bringing down the "big" inefficiency: soon, drawing the entire screen of falling notes (with labels, off-screen indicators, etc.) is going to take a single GPU command, which will be ideal.

Odds and Ends

In no particular order:
  • The next preview will include a 64-bit build for Android.
  • I removed the in-app "Tutorial" screen in the mobile version. Based on how many "the keys are too small!" one-star reviews Synthesia gets on Google Play, it was clear everyone skipped it anyway. That role is better served by additional guides on the support page here on the site. (Then users can auto-translate it into their language of choice, too!)
  • Added a "Restart Loop" shortcut to quickly jump back to the beginning of your loop.
  • Moved the "Restart Loop after X Errors" slider from the Settings screen right to the play screen (while in Loop Edit mode), so it's more convenient and can be changed without quitting the song.
A Very Simple Six Track Recorder

Maybe having any song creation ability whatsoever will quell the requests?

To cause as small an interruption as possible, I scoped this very, very small. There are no editing features. Basically you hit record, play some notes on your keyboard, and when you press stop that's what you get. Beyond layering a few tracks on top of each other, you'll be able to export as a MIDI file and play it like any other... and that's it. The hard parts are all finished and now I'm working on getting the UI bits connected together.

Watching it as it wraps up, I'm starting to get a little worried it might be too simple to be useful. Rather than shelve it at the last minute, I'll at least include the feature in the next 10.6 dev preview so you guys can tell me what you think.

Real Life

We lost a beloved family pet that had been with us almost exactly as long as Synthesia has existed. Between vet visits leading up to and the ennui afterward, last month was a hard time that sapped a lot out of us.

And finally, I'm getting this post out within a few days of my wife's due date on our second child. So, it's probably going to be a busy time for a little while again. I'll try to keep up with the forum and email. I may even be able to sneak some work in during long nights with an infant (periodically) sleeping in a bassinet next to me. :D
Posts: 199

Post by monkel »

Great to read the updates, thanks for sharing.

Had an Uh-oh moment when reading about the recorder. Not sure if it's one those good intentions with a bad aftermath. Hoping for the best!

And all the best for your second one, hope all goes well and you have a great time getting to know each other!
Posts: 13166

Post by Nicholas »

I'm starting to think "Create a Song" deserves to be its own button right on the title screen, instead of this thing that is currently (and clumsily) bolted onto the Free Play screen. That may be how things end up in the long run. (And of course everyone is going to ask to be able to select, nudge, delete, and do every other kind of thing to the recorded notes!) :lol:

As a bonus: I forgot to mention I had a chance to tinker with the first bits of the "no more data loss or corruption" database stuff. I wanted to make sure that you could leave the Free Play screen at any time and not lose any progress on your recording, so it's all saved nearly in real-time. Even a crash on that screen shouldn't impact anything. That's an experimental place to try it first before one of these new smaller releases focuses exclusively on migrating all the data over to that style (plus one sheet music improvement :D ).

Oh yeah: and some more optimization work. I built a neat tool that runs in the debug/development version on my machine that can hook into the Windows memory manager and count how many times Synthesia (or anything that Synthesia uses!) asks for memory. In proper game development, "allocating memory" each frame is frowned on because it can sometimes take an extra long time for Windows to dredge some up for you. (When you've only got 16ms to do all of your work and drawing, an unknown, sometimes-long time spent doing anything is bad!)

I was happy to see that a few of the screens were already at 0 allocations per frame. Then I looked at the play screen... which was a little out of control. The baseline there was ~1200 requests (again, per frame) and it would increase by ~12 or so per falling note(!) visible on the screen. :shock:

Putting that together: in a torture test MIDI with several hundred notes on the screen at once, you could get up above 10k allocations per frame, or more than a whopping half million per second. Our computers are fast these days, but sheesh... what a waste of energy!

Now that the tool exists, this number will be quickly dropping to zero. I would be willing to bet that's the source of most (all?) of the little frame hiccups that show up here-and-there.
Posts: 25

Post by Magic73 »

Nicholas, have you think to use some engine like Unity or Unreal? (this would solve/reduce any multiplatform issue and graphics too)
Posts: 13166

Post by Nicholas »

There was just a discussion about exactly this subject over here.

A condensed summary: there aren't many multi-platform issues these days and the amount of effort it would take to switch would be large.

(All of the various optimization work and other things I described a couple posts up would have to be done no matter the engine that was running things underneath. That stuff comes down to me making naive decisions ~12 years ago when I was only a few years out of college and didn't know what I was doing yet.) :lol:
Posts: 15

Post by spstarr »

Let's *not* think about UE or those graphics engines, Synthesia works fine in Wine when i'm in Linux...
Posts: 12

Post by Fr3dY »

Nice, it's always good to read those development updates! I'm sorry for your pet but wish you all the best for your child!
I guess you'll be quite occupied these days... we'll just wait patiently :)
Posts: 13166

Post by Nicholas »

The quickest update: we're home from the hospital; everyone is happy and healthy. :)
Post Reply