Battlefield 4 Mantle Vs DirectX Benchmark Performance Comparison Milos Milosevic February 18, 2014 Benchmarks Those of you who are expecting to see one of those generic Battlefield 4 Mantle benchmarks tested in a single player environment saying how Mantle brings a 45% increase in frames per second and where everything works smoothly, should be aware that you won’t find something like that here. As we have been saying since the first benchmark we have done, you can’t test a mostly MP game in SP surroundings and expect to receive any meaningful data. Especially when it comes to something as big as a completely new API. This benchmark has been done in all of the settings with a representative amount of CPUs and, at the end, completed with an image comparison, since we believe that nothing should be left unchecked before giving a final conclusion about something as complex as a new DirectX 11.1 alternative called Mantle. We first must apologize to all of our viewers who had expected the Mantle benchmark to be published much earlier. Unfortunately, despite all of our efforts, the Mantle Benchmark had to be postponed due to a large amount of anomalies that we noticed during our testing. One more thing that didn’t help us at all in doing this benchmark earlier was the fact that we hadn’t received the Catalyst 14.1 Beta driver as the rest of the press did, which was 24 hours earlier. When we say anomalies we are not talking about crashing to desktop or BSODs (and we had those in abundance), nor even sudden frame drops, which is something that everyone who has tried Mantle has experienced until now. We had great problems when it came to frame rates on different CPUs and on different settings. Although we reinstalled Windows 8.1, reinstalled Battlefield 4 and installed only the necessary software since we wanted to rule out any possible leftovers from the previous OS that might cause collusion with Mantle, we still had problems when it came to inconsistency of FPS in various settings. For example while benching Mantle after some time we started getting 12 FPS on whatever hardware or setting we tested. We also had huge anomalies when it came to i3 performance since we actually recorded data which showed the i3 beating the i7 and i5 by 30% or more, which absolutely made no sense. Restarting the game, restarting the PC, reinstalling the driver and repairing the game didn’t help at all and just after some time everything started to even out when we realized that Mantle isn’t finished yet and is still in its developing stages, therefore being highly unstable. At the end, we started getting paranoid and reinstalling the driver after every setting change, just in case. This meant that we needed to perform even more testing in order to have somewhat relevant data which of course at the end meant that we needed to postpone the release of our benchmark. The results you see here are the results that you will experience in around 60% of situations, while in the rest of the cases you might receive other anomalies, like the ones mentioned above or maybe something completely different. What is Mantle? Let’s first establish what Mantle is. As you already know, Mantle is a new API developed by AMD. This company and many other game developers have been complaining for years how DirectX is actually limiting the real potential of modern GPUs and CPUs. As a result of that, AMD has been developing this new API for quite some time now, which will allow developers to have low access to hardware. The same low access, incidentally, that developers have when it comes to console game development. We all know that AMD’s CPUs and GPUs are being put into today’s Playstation 4 and Xbox One, so after they made the API for consoles it made sense to create an API for the desktop sector. According to the parties of interest, the biggest bottleneck when it comes to DirectX is in draw calls. To put it simply, draw calls are materials being drawn on the screen. Using DirectX an average game developer can achieve around 3000-5000 draw calls. Developers who invest more time into optimization can get around 10,000 draw calls, but with Mantle, according to AMD, developers will be able to achieve closer to 100,000 draw calls and that’s only the start. With large amounts of available draw calls developers can focus on achieving more realistic models and surroundings, or on the other hand focus on performance which will enable users with less powerful PCs to play very demanding games at acceptable frame rates. Since Mantle will enable more communication between applications and GPUs, this will significantly reduce the CPU load which will give more power to developers in achieving better performance on the same hardware than with DirectX. Of course this is all in theory but let us see how Mantle actually performs in real life situations. Testing Components and Methodology For benching Mantle we used our standard CPU pallet consisting of : i7 4770k i5 4670k i3 4340 FX 8350 FX 6300 FX 4300 Although AMD claimed that all of the GPUs that feature a GCN architecture are supported by Mantle we decided to play it safe and use only the R9-290. Of course we used the Catalyst 14.1 Beta driver. When it comes to the testing methodology we used our standard scenario where we performed testing on an empty server, so bear in mind that the frames per second that you will see on the graphs will vary once you are on the full 64 player server. As we explained before in our Battlefield 4 Win7 Vs Win8 Benchmark this kind of scenario is necessary because it is impossible to get steady frame rate while there are enemy players all around you. With so many variables and with killcams ruining your FPS it is impossible to make a relevant benchmark in such conditions. Therefore, this benchmark can only be used to measure the performance difference between CPUs and not as the actual in-game performance, because as always it might and it probably will vary. And before you ask, when switching to Mantle or back to DirectX 11.1 we always restarted the game and as we mentioned in the introduction we even went so far as to restart the PC and even reinstalling drivers over and over again. All of the frametimes, CPU frametimes and GPU frametimes have been measured using the in-game console command “PerfOverlay.FrameFileLogEnable 1” and we later calculated the FPS using the same data. This has been used while benchmarking Mantle (since there is no other way) and in order for it to be a fair comparison we used the exact same method while benching the game in DirectX 11.1 mode. Those of you who are haven’t seen our benchmarks till now, grab your popcorn and prepare to be overwhelmed. 2560×1440 Battlefield 4 Ultra Mantle Vs DirectX 11.1 CPU Benchmark At 2560×1440 we don’t see the real benefits from Mantle, at least when it comes to Intel’s CPUs. Haswell CPUs perform almost identically on both APIs apart from the i3 which actually performs worse while running on Mantle. When it comes to AMD’s FX processors the situation is much more complex. We do see an increase of around 40% in average and max FPS but we also see a downfall at min FPS of around 45-50%. This goes for the FX 8350 and FX 4300 processors while the FX6300, although performing better than any other AMD CPU from our test bench in DX 11.1 mode, performs worse when we switch to Mantle. 2560×1440 Ultra Battlefield 4 Mantle Vs DirectX 11.1 Frametimes Frametimes tell their own story, and although variation is less with Mantle these huge spikes really kill it when it comes to a smooth game experience. Notice that the FX 8350 and FX 4300 behave almost identically either in DirectX 11.1 or Mantle, while the FX 6300 looks like it has been forgotten by AMD and DICE when they wrote the code. At the highest possible settings the CPU doesn’t play a major role when it comes to frames per second (probably because there aren’t that many of them) that’s why Intel’s i7, i5 and i3 perform similarly. Of course once we lower the settings the difference will increase. 2560×1440 Ultra Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes 2560×1440 Ultra Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes 2560×1440, High Battlefield 4 Mantle Vs DirectX 11.1 CPU Benchmark Those of you who have already seen our BF4 benchmarks know that the i7 4770k and i5 4670k perform, in a best case scenario, the same. The i7 at some point will actually provide less FPS than the i5 4670k itself. Back then we came to the conclusion that this comes to the low work load. The game’s engine is optimized to deliver a full load for a 4 thread CPU in Intel’s case and a 6 thread CPU in AMD’s case. That is the reason why the i5 4670k performs better than the i7 and why the FX 6300 performs better than the FX 8350. This does not mean that the game does not use all CPU cores and threads in i7’s and FX 8350 case, it means that there appears to be some kind of a lag in communication between those cores which results with lower frames per second than expected. Now we see that with Mantle things kind of stay the same when it comes to work load. The only exception is that the FX 6300 performs sluggishly with the new API. We also see those huge frame rate drops that everyone has experienced with Mantle. It is hard to say in which situations this exactly happens but this reminds us of the poor performance of AMD’s FX 6300 and FX 4300 when Battlefield 4 was still in Beta, although the reason for these frame drops here might be entirely different. Although Mantle does bring more FPS in max and on average, it completely fails when it comes to min FPS which is the most important thing for a first person shooter game. 2560×1440 High Battlefield 4 Mantle Vs DirectX 11.1 Frametimes On high settings we have more or less the same ratio for all of the CPUs as with Ultra settings . Intel’s i7 and i5 perform well with Direct X 11.1 and with Mantle (with some frame drops here and there), while the i3 continues to underperform with Mantle. The FX 8350 and FX 4300 get a nice boost with Mantle but fail in min FPS while the FX 6300 fails in all departments performing even worse than with DirectX 11.1 2560×1440 High Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes Once again we see huge spikes with AMD processors, either with Mantle or with DirectX 11.1. Frametime variance with Intel’s CPUs are constant and stable, however we see that the less powerful i3 again loses performance when Mantle is enabled. 2560×1440 High Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes Again it’s quite interesting to see that R0 290’s GPU frametimes are somewhat good with AMD’s FX 8350 and FX 4300 while the FX 6300s chart looks like a chart from a patient who is about to have a cardiac arrest. 2560×1440, Medium Battlefield 4 Mantle VS DirectX 11.1 CPU Benchmark On lower settings we get an increment in frame count, and this is the point where processors tend to distinguish a bit more. Or do they? The i7 performance is pretty much the same in both cases while the i5 gets a small boost of around 10% with Mantle. The i3 continues to do well under DirectX 11.1 and continues to lack under Mantle. But the real show begins with the FX 8350 which gets an 80% boost in average and max FPS with Mantle, but again sudden frame drops are taking their toll on the CPU. Although it’s worth mentioning with Mantle the FX 8350 has actually managed to catch up with the i5 and i7 in average fps and even beating both in max FPS, as we said before min FPS just kills it. A similar story goes for the FX 6300, which continues to perform badly under Mantle and we are really not sure why DICE and AMD have completely ignored optimization for this CPU since it performed better out of all the FX processors under DirectX 11.1 2560×1440 Medium Battlefield 4 Mantle Vs DirectX 11.1 Frametimes Frametime variance is something that is really noticeable when you compare Mantle and DirectX 11.1 in this case. AMD’s FX CPUs frametime variance is actually less with DirectX 11.1 than with Mantle although frame rates are slightly better. 2560×1440 Medium Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes Just one more example of horrible frametime variance with Mantle. The graphs are just simply all over the place and it is quite impossible to see the i5 and i7 graphs due to the oscillation of FX CPUs which is quite overwhelming. 2560×1440 Medium Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes The R9 290 frametimes are quite constant at these settings no matter what API you use, but look at the frametimes that the R9 290 produces with the FX 8350 under Mantle. These are the best GPU frametimes we have seen so far although the variance is not as good as with Intel’s i5 and i7 processors. 2560×1440, Low Battlefield 4 Mantle VS DirectX 11.1 CPU Benchmark At the last setting for 2560×1440 things more or less stay the same. The i7 and i5 perform the best out of all of the CPUs where again the i5 dominates. The i5 also gets a slight boost with Mantle of around 10% while the i7 gets a 5% increment. The i3 does well with DirectX 11.1 but again not really able at least to duplicate the result with Mantle. The FX 6300 is sharing the same fate with the i3 and we can easily say that the pattern of halved performance is beginning to be quite visible. The FX 8350 and FX 4300 get the usual boost followed by the horrifying frame drops. 2560×1440 Low Battlefield 4 Mantle Vs DirectX 11.1 Frametimes Nothing new to see here, while overall frametimes are more or less constant with DirectX 11.1 with Mantle they just go through the roof. The i7 and i5 remain in providing stable performance throughout the whole benchmark and with all of the APIs while the rest of the CPUs are struggling. The FX 4300 continues to provide the worst frame variance and the FX 8350 is slightly behind with the same anomalies. 2560×1440 Low Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes 2560×1440 Low Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes Again we see here how much Mantle, we can say it now with a considerable amount of certainty, penalizes the R9 290 when paired with the FX 6300 and Intel’s i3 4340. The i7 and i5 remain immune to any API while once again with the FX 8350 and with FX 4300 GPU frametimes are more or less the same. Of course huge spikes with the FX 4300 are unavoidable. 1920×1080, Ultra, Battlefield 4 Mantle VS DirectX 11.1 CPU Benchmark As we step down to 1080p CPUs are performing in more or less an expected manner. The i7 and i5 are still kings of the hill with both DirectX 11.1 and Mantle. The i3 performs well under DirectX but again frames drop again once you start benching with Mantle. Frame drops which have been so prevalent with FX CPUs are plaguing their performance once gain, either with DirectX or with Mantle where with Mantle, where performance drops are more visible. We once again see the FX 8350 almost reaching i7 and i5 performance with Mantle (FX 4300 as well). 1920×1080 Ultra Battlefield 4 Mantle Vs DirectX 11.1 Frametimes Overall frametimes are more or less what we have seen before, but we see in this scenario that the FX 6300, although performing better than all the other FX processors until now, gets beaten by the FX 4300. Frametime variance under DirectX is pretty bad with AMD CPUs while Intel’s processors are a completely different situation. With Mantle we again see a pattern from the previous benchmarks, huge spikes with FX processors with somewhat ironed out frametimes. 1920×1080 Ultra Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes Here we see AMD’s FX 6300 getting severely penalized using Mantle, with spikes so high that we had to cut the graph in order for other CPUs to be visible. This is a clear lack of optimization, which is strange because the FX 6300 is a recommended CPU for the game by DICE. While it runs very well under DirectX 11.1 one would think that optimization for this CPU for Mantle should be superb. But as you can see it isn’t, far from it. 1920×1080 Ultra Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes There is a clear miscommunication between FX processors and the R9 290 here. A page or two back, we saw the R9 290 with better GPU frametimes with the FX 8350 out of all of the CPUs so it is possible to have stable GPU frame rates with AMD’s processors. However, we will obviously have to wait for the future improvements because now this doesn’t look like anything worth bother with. 1920×1080, High, Battlefield 4 Mantle VS DirectX 11.1 CPU Benchmark When announcing the Catalyst 14.1 Beta AMD had stated that the new driver will mostly bring benefits to slower CPUs, while those with more horsepower should get an increment of around 10%. This appears to be true, at least to some degree. The i7 and i5 do get a boost of around 10% on average with Mantle, but the i3 doesn’t get any improvement in terms of performance. As a matter of fact it’s quite the opposite. We have already mentioned at the beginning of the article about the anomaly that happened to us where the i3 CPU was beating the i7 and i5 processors in all settings and resolutions by around 30% on average. We had no explanation for what happened but after a while it all went back to normal. Unfortunately we haven’t made screenshots when this happened so we are unable to compare image quality with the ones we created later. Of course if this does happen again we will conduct a full investigation. The issue with FX processors and Mantle continues where neither of them get acceptable and stable frame rate. The rest of the graphs below tell the story told so many times. 1920×1080 High Battlefield 4 Mantle Vs DirectX 11.1 Frametimes 1920×1080 High Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes 1920×1080 High Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes 1920×1080, Medium, Battlefield 4 Mantle VS DirectX 11.1 CPU Benchmark At these settings we see the first change in the usual pattern. Apparently the i3 4340 is getting a boost with Mantle for the very first time, so those of you who want to experiment with Mantle and have this CPU should try first at these settings. Also the FX 6300 finally gets its boost of around 10% with Mantle while the FX 8350 gains around a 40-45% increase. Sudden and huge frame drops come with the package. We also see how the i5 beats the i7 under DirectX 11.1 while with Mantle the i7 manages to catch up and eventually surpass it, although i5 gets a slight increment in performance as well. 1920×1080 Medium Battlefield 4 Mantle Vs DirectX 11.1 Frametimes If there is anything here that is different from our previous charts, it’s the i5 4670k frametimes being better than the i7. Until now you could hardly tell those two apart in the graphs but we see a separation when it comes to performance. This reminds us of our Battlefield 4 W7 vs W8 benchmark where the i5 was beating the i7 in all settings and only under W8 it managed to get somewhat closer to the i5. We see almost the exact same situation here with DirectX 11.1 and Mantle. This goes for CPU frametimes and GPU frametimes as well. 1920×1080 Medium Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes 1920×1080 Medium Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes 1920×1080, Low, Battlefield 4 Mantle VS DirectX 11.1 CPU Benchmark On our final benchmark setting we see that not even Intel is immune to huge frame drops under Mantle. The i5 4670k gets a min FPS drop to only 4 while the i7 4770k has a drop in performance, though not as drastic as the i5. Results that you see in this benchmark are repeated results that we gathered after 5-6 tests but we can say with some level of certainty that you will probably get frame drops with any CPU and on any settings as long as you use Mantle. AMD’s CPUs are hiccupping with every API on this setting. Here we see all the bad things that we saw throughout the whole benchmark all placed together. 1920×1080 Low Battlefield 4 Mantle Vs DirectX 11.1 Frametimes 1920×1080 Low Battlefield 4 Mantle Vs DirectX 11.1 CPU Frametimes 1920×1080 Low Battlefield 4 Mantle Vs DirectX 11.1 GPU Frametimes Battlefield 4 Mantle VS DirectX 11.1 Image Quality Comparison Conclusion Before we jump to the final conclusion let’s see how Mantle uses each CPU differently than DirectX 11.1 We see that when Mantle is enabled the game employs quad core/thread CPUs with one thread extra. All of you can see it, all you need to do is to press console key “~” and type: “Render.DrawScreenInfo 1” Don’t be confused if you see 8 cores and you have an i7 4770k, which is a quad core but since it has Hyper-Threading enabled the operating system sees it as an eight core CPU, therefore the game as well. There is one more thing that DICE implemented with the latest patch, and that is command (or at least they say they did) which should enable a user to configure and control how many CPU cores should be used in the game with the new “Thread.MaxProcessorCount” settings in User.cfg. We tried this numerous times but it just refused to work. We also tried typing it into the console but again it failed. So this feature isn’t working at all. Now from what we managed to see, Mantle is not really working and its currently still in its Alpha stages so we can’t make any prognosis about the future of this alternative API, but we can say that at this point in time that Mantle is not good enough for you to want it and use it. Lots of issues that FX processors have with DirectX have been transferred to Mantle and some of them received additional problems. The reality isn’t that nice and fluffy as we were led to believe, but than again most of you have found out by now what AMD brought to the table with Mantle. The fundamental problem with Battlefield 4 is CPU workload and this is why slower CPUs with less cores or threads perform better than the more powerful ones. As long as it’s low, no Mantle and extra threads in the world will help you. We saw some Mantle benefits for the i7 but nothing too conclusive, and the same thing goes for the FX 8350. If you remember, when Mantle was enabled, the FX 8350 was getting a considerable boost but so was the FX 4300. For most of the benchmark the FX 8350 and FX 4300 went hand-in-hand in terms of performance while the FX 6300 was mostly getting the bad end of the stick, and has recovered only with the last two benchmark settings. Image quality with Mantle is also debatable, and we cannot say that it’s the same as with DirectX 11.1 and for the time being we also can’t make any comments on this issue because Mantle is far from a finished product. DICE needs to further improve the performance of multicore processors by increasing the workload. For the time being Mantle is just a glimpse of what we may get in the future. How far that future will be is a totally different question. See Pricing If you have any questions about the benchmark please ask in the comment section below and I will be glad to answer.