Larger view of FPS results from MSI Afterburner, plus additional calc's. I did the benchmarking on an Nvidia GTX760.
There's 2 things important for FPS:
1) how high they go
2) how consistent they stay (ie: no frame drops, lag, etc)
MSI Afterburner gives us the Avg, Min, Max & Lows, and those are good to see how high the FPS get, and sort of ok to tell how consistent the frames are staying.
But, there's a better way to measure FPS consistency / smoothness: Coefficient of Variation.
CV in statistics is how tightly plot points hug the average line. You can use this for process analysis and optimization, because plot points tightly hugging the avg are more consistent, which means a process is more in control. Meanwhile, plot points all over the place are showing massive inconsistency in performance, and thus a process out of control.
("Control" in this sense is simply a process' ability to provide a consistent output at each plot point.)
When we look at the Lows, the 0.1% low is actually the 3rd standard deviation on a bell curve. Edwards Deming (Total Quality Management / TQM guru) said that things going beyond the 3rd standard deviation are special plot points going outside the normal process' control, and need to get looked into.
But, what we can do here is use some TQM / Six Sigma Black Belt / Project Management techniques to estimate CV, and by extension, FPS smoothness.
CV = Standard Deviation / Average
We already have the Avg, but we need the standard deviation.
Standard Deviation has a formula to generate an actual SD, but there's an estimation formula used in Project Management to quickly determine it...
Std Dev Estimate = Range / 4
We get the Range by subtracting the Max FPS from Min FPS...
So...
Range = Max - Min
Std Dev = Range / 4
CV = Std Dev / Avg
... or ...
CV = ((Max - Min) / 4) / Avg
If we didn't have the Max or Min, we could even estimate the Range by using the Avg - 0.1% low and multiply that by 2 (b/c Avg - 0.1% low would be equal to 1/2 the bell curve, so half the range).
Anyways.. we have our estimates.. so we can calculate CV. But, CV tracks as an inverse %, IE: lower is better. This is confusing to most folks, because many folks are conditioned to think higher is better. So, we can invert it with 1 - CV and call it "smoothness" or "efficiency".. which makes much more sense to a layman.
So, our table above shows the "<-- Important" spots...
1) FPS Avg shows how high our FPS is normally tracking
2) FPS Smoothness is showing how consistent our FPS
You want both to be high, ideally.
But, FUEL throws us a ringer. Asobo tied in-game performance of vehicles / AI / etc to the FPS. So, if you uncap FPS (the way I did here to get max FPS analysis via turning off VSync in Nvidia Control Panel), the game starts to go wonky. The AI vehicles start to speed off so fast in the distance you'll never catch them in races.
So, FUEL needs to get capped at 60 FPS.
So, what was the point of this analysis?
1) we want to see if the changes made to the new shaders impact the FPS in a positive or negative way.. since the performance is pretty near the original shaders, it means the features added didn't impact performance much while adding value to the end result
2) we want to see if we're maxing out our 60 FPS cap as much as possible, because if we are, then we're maxing FPS and maxing smoothness more.
From the benchmarking we can see that our new shaders don't impact performance much. We don't have massive gains, but we don't have massive hits, either. Likewise, we see that we're pretty consistently hitting 60+ FPS, and our "smoothness" is in the mid 80%'s. That's not bad. The smoothness is impacted by how many objects are on the screen, etc. (Fewer things to process, higher FPS. More things to process, lower FPS). A smoothness in the 90%'s would be fantastic. FUEL has some frame drops every now and then, but looking at the Min FPS, it's just down to 60's or 50's.. it's not like it's tanking into the 10's or such (which would be massive frame drops.)
So, the new shaders tweak visuals without impacting performance much.
Larger view of FPS results from MSI Afterburner, plus additional calc's. I did the benchmarking on an Nvidia GTX760.
There's 2 things important for FPS:
1) how high they go
2) how consistent they stay (ie: no frame drops, lag, etc)
MSI Afterburner gives us the Avg, Min, Max & Lows, and those are good to see how high the FPS get, and sort of ok to tell how consistent the frames are staying.
But, there's a better way to measure FPS consistency / smoothness: Coefficient of Variation.
CV in statistics is how tightly plot points hug the average line. You can use this for process analysis and optimization, because plot points tightly hugging the avg are more consistent, which means a process is more in control. Meanwhile, plot points all over the place are showing massive inconsistency in performance, and thus a process out of control.
("Control" in this sense is simply a process' ability to provide a consistent output at each plot point.)
When we look at the Lows, the 0.1% low is actually the 3rd standard deviation on a bell curve. Edwards Deming (Total Quality Management / TQM guru) said that things going beyond the 3rd standard deviation are special plot points going outside the normal process' control, and need to get looked into.
But, what we can do here is use some TQM / Six Sigma Black Belt / Project Management techniques to estimate CV, and by extension, FPS smoothness.
CV = Standard Deviation / Average
We already have the Avg, but we need the standard deviation.
Standard Deviation has a formula to generate an actual SD, but there's an estimation formula used in Project Management to quickly determine it...
Std Dev Estimate = Range / 4
We get the Range by subtracting the Max FPS from Min FPS...
So...
Range = Max - Min
Std Dev = Range / 4
CV = Std Dev / Avg
... or ...
CV = ((Max - Min) / 4) / Avg
If we didn't have the Max or Min, we could even estimate the Range by using the Avg - 0.1% low and multiply that by 2 (b/c Avg - 0.1% low would be equal to 1/2 the bell curve, so half the range).
Anyways.. we have our estimates.. so we can calculate CV. But, CV tracks as an inverse %, IE: lower is better. This is confusing to most folks, because many folks are conditioned to think higher is better. So, we can invert it with 1 - CV and call it "smoothness" or "efficiency".. which makes much more sense to a layman.
So, our table above shows the "<-- Important" spots...
1) FPS Avg shows how high our FPS is normally tracking
2) FPS Smoothness is showing how consistent our FPS
You want both to be high, ideally.
But, FUEL throws us a ringer. Asobo tied in-game performance of vehicles / AI / etc to the FPS. So, if you uncap FPS (the way I did here to get max FPS analysis via turning off VSync in Nvidia Control Panel), the game starts to go wonky. The AI vehicles start to speed off so fast in the distance you'll never catch them in races.
So, FUEL needs to get capped at 60 FPS.
So, what was the point of this analysis?
1) we want to see if the changes made to the new shaders impact the FPS in a positive or negative way.. since the performance is pretty near the original shaders, it means the features added didn't impact performance much while adding value to the end result
2) we want to see if we're maxing out our 60 FPS cap as much as possible, because if we are, then we're maxing FPS and maxing smoothness more.
From the benchmarking we can see that our new shaders don't impact performance much. We don't have massive gains, but we don't have massive hits, either. Likewise, we see that we're pretty consistently hitting 60+ FPS, and our "smoothness" is in the mid 80%'s. That's not bad. The smoothness is impacted by how many objects are on the screen, etc. (Fewer things to process, higher FPS. More things to process, lower FPS). A smoothness in the 90%'s would be fantastic. FUEL has some frame drops every now and then, but looking at the Min FPS, it's just down to 60's or 50's.. it's not like it's tanking into the 10's or such (which would be massive frame drops.)
So, the new shaders tweak visuals without impacting performance much.