NB: Make sure you read the whole text and not just look at the results in the picture. There are some important findings in my testing that is not visible in the attached picture.


Hi,

I have had some time off these past few days, and I've dedicated some of it to a little test. I've based it on a little test that is available for download from http://www.dawbench.com/benchmarks.htm

I'm in the market for a new computer, and I decided to test my current system so I can have something to compare the alternatives to. I also decided to compare the performance of Studio One to Reaper.

Apple macOS has hyperthreading enabled by default. By downloading a package of developer tools for macOS, I've been able to turn off hyperthreading on the OS level. I wanted to check the performance of the DAW both with hyperthreading enabled and with hyperthreading disabled. Studio One has the ability to switch off virtual cores on a DAW level. I decided to test this as well, i.e. with hyperthreading enabled in the OS and with 4 cores disabled in the DAW.

The computer used is a 2012 Apple MacBook Pro 15" retina 2.6 GHz with 16 GB RAM and 512 GB SSD running Studio One 3.3.3 on Mac OS X 10.9.5 (Mavericks). The audio interface is a Lynx Hilo connected via USB 2.0.

Basically, I've done two different tests.

1. One is the standard DAW Bench test made for Studio One and Waves Kramer Tape. The test consists of 4 x stereo tracks of audio content, 1 x stereo track called Sine Monitor, 40 x stereo tracks of sine waves and 320 x inactive Kramer Tape insert plugins (8 per sine wave track) to be turned on one at a time until the session is overloaded and clicks and pops occur.

2. The other test is based on test no. 1, except I decided to load several instances of Cytomic's The Glue (with 2X over sampling) and all of them onto the same Channel, from the track called Sine Monitor. The reason for this, is because Studio One is unable to distribute the load of a single channel across multiple cores, so I wanted to test how many instances I could load onto a single Channel with and without hyperthreading. I also wanted to compare Studio One's behavior to Reaper's.

Here are the results (the numbers indicate the number of instances of the plugin I could run without hearing any pops or clicks; f.i. I could run 62 instances of The Glue on one single-channel in Studio One when using a 4096 buffer and with hyperthreading disabled in macOS):



As you can see, some of my findings are quite surprising. Especially the buffer of 2048 seems to act weird in Studio One on my system. Also, Reaper gives very, very different numbers on lower buffer settings compared to Studio One, but on higher buffer settings the performance seems closer and closer until Studio One actually marginally outperforms Reaper in certain situations (single-channel performance with the two highest buffer settings and hyperthreading disabled on OS level). I chose to write 0 plugins instances for the 32 buffer, but none of these DAWs could play back the project without clicks and pops when running my system on a 32 samples buffer, even though I didn't load any plugins. Another thing to notice, is that disabling cores in Studio One's options is not the same as disabling hyperthreading on an OS level. The latter seems to be a lot more efficient on my system when comparing the two; however, none of them can match having hyperthreading enabled when confronted with the multi-channel test. Reaper behaves differently. Studio One seems to be a lot more "picky" as to what settings I use, while Reaper's performance results are surprisingly consistent, no matter what settings I use. In Reaper, the differences between the different buffer settings and between the two hyperthreading settings are remarkably small compared to Studio One, but also here hyperthreading is more efficient for the multi-channel test, while disabling hyperthreading can be marginally more efficient also in Reaper when using smaller buffer settings in the single-channel test. All in all, when it comes to number of plugin instances, Reaper is quite a bit more efficient than Studio One on my system, especially when using lower buffer settings.

However, this was only the beginning of my testing. I also decided to test other behaviors of these DAWs. Here are some other very interesting findings:


Core distribution for heavily loaded single-channels:
Reaper seems to be able to spread all the plugins of that single track across multiple cores. In the OS's Activity Monitor (set to refresh displayed info every second) it shows an interesting behavior, where the load seems to alternate between the cores. One second it is perhaps 60% taxing core 1 and just around 10% for the other cores. And then, a second later, it is core 2 that is taxed 60% and the rest is around 10 %, and the next second it is core 1 and 3 that both are taxed around 30-40% each and the other two are around 10%, and the next second it is core 4 being taxed 70% and the rest around 10% and so on. It alternates. In Studio One it is pretty much all on core 1 when dealing with that single track.

Core distribution for heavily loaded busses:
One might think that a workaround for having more plugins on one track could be to route the track/channel to an intermediate bus and putting the required plugins there. Apple suggests this workaround for Logic in this article: https://support.apple.com/en-us/HT201838. This does not work in Studio One nor in Reaper. A continuation of the signal in this way will, as far as processing distribution, make it behave like it is the same track, and there is no difference in how many plugins you can load doing it this way compared to having it all on one track/channel. (Reaper might give you a mere 0.5% more efficiency routing it via such an intermediate bus, but my test was not really conclusive here and it was, if any, next to nothing in difference.)

Core distribution for heavily loaded master bus:
Interestingly, Studio One treats the master bus the same way as it does regular busses. This means that if the core is maxed out in the track/channel, you can not expect to load anything extra on the master bus. Reaper, however, behaves very differently. It seems to treat the master bus as an entirely separate thing and even though I had fully maxed out the single track, I could load an extraordinary amount of additional instances of the plugin onto the master track. Very different behavior compared to Studio One and one of the most interesting findings in this test.

Sends:
Studio One appears to put the signal going to a send on the same core as the track I’m sending from. This means that if I have a track/channel loaded to the max, I will not be able to send from this track to a bus/FX channel with a plugin on it. I could send other tracks to the bus/FX channel, but not the maxed out track/channel. It uses the same CPU core. Reaper behaves similarly in that you cannot expect to send from an already maxed out single-channel. However, and this is a very, very interesting point, in Studio One, setting up a send from your maxed out track/channel and muting it still acts as if I’m actually sending signal to the bus/FX channel. That button is apparently not an on/off switch but a simple mute, so even though I’m "switching off” the send (f.i. on the maxed out single track) and no audible signal is routed to the corresponding bus/FX channel, the CPU core still behaves as if it’s on. This is very important to know about: This is a mute, not an on/off switch. If you want to save CPU power by not sending to your reverbs f.i., it is not enough to turn off the send by pressing the blue button to its right, you actually have to remove the whole send (I’m not talking about removing the return, i.e. the bus/FX channel and its plugins, but I’m talking about removing the send instance from the track/channel you’re sending from). Muting it won’t help you save CPU power for your maxed out core. (And … I don’t know if any of you work like this … but if you use an old school approach where you have a template to mimic how an analog console workflow used to be, where you had sends for every track on the console and they were always set up to the same rack FX, then you can imagine how much extra cpu you’re using for having those unenabled sends just sitting there waiting for you to dial them in, audible idle, but still engaged CPU-wise.) Oh, and by the way, Reaper does not do this when it comes to muting the sends. The M-button for Reaper's sends looks like a mute, but actually is an on/off switch.

Latency:
Studio One feels responsive also when loaded heavily. Reaper, on the other hand, behaves quite differently. With higher buffer settings it takes a while before it starts up after pressing play. This feels really weird after having used Studio One for a long time now. This latency can be very annoying also when mixing. With my most extreme loads we’re talking up to several seconds of latency before playing back (depending on what’s on the tracks and the master bus). This lack of responsiveness also goes for stopping the Reaper project. The playback cursor and the time/bar counter stops immediately after pressing stop, but the tracks in Reaper continue to play after you’ve stopped. And, even more interesting, the track with the heavy load of plugins is naturally more delay compensated, so in Reaper (at least in my test) this makes the normal audio tracks have one amount of latency while the single heavily taxed track has a longer latency. In other words, the test track with a lot of plugins continues to sound out of the speakers relatively long after the other tracks have stopped. With a heavy load on the master channel and a heavy load on one single channel but nothing on the others, I got some results with huge latencies where start of playback was delayed by 4 seconds, stopping of unprocessed single tracks were delayed by around 3-4 seconds and stopping of the single heavily loaded track was delayed by around 7 seconds (an additional 3-4 seconds compared to the other tracks). It sounds really weird hearing that single track playing all by itself for some seconds after the others have stopped. In total, this is an annoying behavior and even though the above settings are quite extreme and highly unlikely in a real world scenario, it kind of tells me why Reaper will let you tax it more heavily, but if you do, it still comes at a cost of responsiveness and latency if the plugins need their own buffer (f.i. for look-ahead, linear phase or over-sampling etc.).



Requests:
I kind of like Reaper, nothing really wrong with it, but nothing compares to Studio One's workflow, which I love. In the next revision of Studio One, I hope there will be some options where we can choose what kind of core distribution behavior we want for intermediate busses and the master bus. Oh, and I hope that the send mute button changes into a true send on/off button.

Final thoughts:
Back in the day, audio engineers used to test their equipment with test signals and what not. Understanding how your tools work and how to use them used to be important. Nowadays, many of us just assume that things work, and if they don’t we want someone else to fix them. No wonder, since the consumer is only provided with closed systems and no one really understands (or cares?) what is actually happening behind the scenes. This goes for pretty much any part of modern life and modern civilisation. If something happens to your modern car, good luck fixing it yourself. And there doesn't seem to be much to do about it, at least I don’t have the time for all that. Even as an audio engineer I don't have the time to be a in-depth computer tech, but a minimum of knowledge of some of my system's inner workings is still needed. These past couple of days I found some time to test my system. I’m glad I did. Now I know more about the tools I use 12 hours a day 6 days a week. Thought I'd share my findings with you guys. Hope you also got something from this.