Rating Estimations between different Hardware

[spacious_mind]

While testing Gavon, Gavon2, Resurrection & Revelation with the rating test, I thought I would use this opportunity to compare the different hardware to see what the speed doubling factors might look like. As a basis for these tests I used CCRL's chess engines Website:

http://computerchess.org.uk/ccrl/4040/r ... t_all.html

This site is the best for comparison because it has the most games and programs all tested under the same conditions using an Athlon 64 X2 4600+ (2.4 GHz).

The below chart is something I created based on using the Athlon 64 X2 4600+ (2.4 GHz) as a start point.

HARDWARE MIPS COMPARISON CHART



The above chart shows the Hardware speed and its rated MIPS. I have added a factor that can be used for speed doubling in order to estimate the rating loss or gain between hardware. For example if it is a Gavon program and an estimated ELO of 70 points is used, then the estimated increase or loss in rating over different hardware would look as follows:

Example Gavon Chess Program X = 2500 ELO

Athlon 64 X2 4600+ (2.4 GHz) = 70 * 3.96 = +277 ELO = 2777 ELO (estimated)
Gavon 2 = 70 * 0.77 = +54 ELO = 2554 ELO (estimated)
Revelation = 70 * 0.35 = -25 ELO = 2475 ELO (estimated)
Resurrection = 70 * 2.19 = -153 ELO = 2347 ELO (estimated)

With this in mind I next compared the results of Resurrection with CCRL's 40/40 site ratings.

RESURRECTION PROGRAMS RESULTS



I could not get a rating for Deep Sjeng 1.8 therefore it had to be excluded in the above comparison.

For Resurrection programs the average factor per doubling is 66 ELO


REVELATION PROGRAMS RESULTS



For Revelation the average factor for the programs I have tested so far is 69 ELO.

GAVON 2 PROGRAMS RESULTS



The ones I have removed are programs that do not show a rating at CCRL. For Gavon 2 the average factor is 69. Obviously there a still a lot of Gavon 2 programs that I have to test.

GAVON PROGRAMS RESULTS



I have tested all the Gavon programs therefore this comparison is the most accurate. The ones in the above chart left blank are programs that are missing at CCRL also the ratings below 2500 ELO at CCRL started to become less accurate therefore I omitted all programs rated below 2500 at CCRL in this comparison.

For Gavon the average factor was 71.

Totaling all the above tests you get the following average for factor doubling:



The average comes out at 69 ELO therefore this does seem to confirm the most commonly used rule of thumb of 70 ELO that most people use. Individually however the programs do vary a lot so therefore this is good as a rule of thumb only.

RESURRECTION TO REVELATION COMPARISON



I have not completed all the Revelation tests but the above table shows that there might be a big performance improvement with Revelation over Resurrection.

GAVON TO GAVON 2 COMPARISON



Same with Gavon 2, I still have to do a lot of rating tests, but the ones I could compare so far show an average of 66 ELO per speed doubling.

FRUIT 2.1 COMPARISON



Fruit 2.1 is the only program common to all tested Hardware. Average for Fruit 2.1 is 64 ELO. With Revelation, Fruit 2.1 must have found it's sweet spot because it slightly scored better than Gavon in the rating tests.

To summarize as a rule of thumb 70 ELO per speed doubling is a reasonable number. But it can vary a lot from program to program and between different hardware therefore it should be used with this knowledge in mind.

Best regards

[dedicate computers]

Nick, absurd the rating of engines running on Gavon, impossible for a human to win in blitz chess. Nick, you worked hard to get these results!
With gavon regards
Oswaldo

[spacious_mind]

Nick, absurd the rating of engines running on Gavon, impossible for a human to win in blitz chess. Nick, you worked hard to get these results!
With gavon regards
Oswaldo

Hi Oswaldo,

Thanks, but it really is not hard work, if I didn't do this I would be playing dedicated chess matches instead.

Believe it or not I have as much fun with these tests as I do playing tournaments. Reason being you have the same excitement when a computer performs well and added to this you can't beat the experience of seeing their good points and weakness when I compare them in these exact same test games.

As for strength yes generally speaking they are too strong as is Resurrection, Revelation, Revelation II & Mysticum. But in all of them there are fortunately a few programs that you can play against or against other dedicated chess computers. The nice thing is I can start planning a top programs dedicated tournament and actually include them all as well as DOS programs and Palm & Pocket PC programs & even Xbox

This really was not very interesting in the past because just playing Resurrection against Revelation is not very interesting because there was not enough program variety really.

I am not a fan of artificially slowing down a top modern engine through internal engine parameters which some engines offer, this was never an option for me unless you do it by slowing down the hardware itself where a program would still play as it would when installed but cannot search as deeply because of the slowdown in hardware, and not the program parameter settings of the program. This is not an acceptable option for me to play tougher engines against dedicates or even against myself. As this is like playing a beginner in chess and he sees you deliberately playing bad moves. A beginner does not have much satisfaction of you treating him in this way. Artificial parameter slow down to me is exactly the same, very condescending.

Maybe I am funny about this because I am sure that many people don't get it, that I am against this.

Anyway that is why I am excited to have all these options nowadays because of Gavon and Mysticum.

Best regards

[spacious_mind]

Here is an update for Revelation. All 11 chess engines have now been tested.

REVELATION PROGRAM RESULTS



Average of 69 ELO per speed doubling remains as before.

Best regards

[ricard60]

Nick, absurd the rating of engines running on Gavon, impossible for a human to win in blitz chess. Nick, you worked hard to get these results!
With gavon regards
Oswaldo

I am not a fan of artificially slowing down a top modern engine through internal engine parameters which some engines offer, this was never an option for me unless you do it by slowing down the hardware itself where a program would still play as it would when installed but cannot search as deeply because of the slowdown in hardware, and not the program parameter settings of the program. This is not an acceptable option for me to play tougher engines against dedicates or even against myself. As this is like playing a beginner in chess and he sees you deliberately playing bad moves. A beginner does not have much satisfaction of you treating him in this way. Artificial parameter slow down to me is exactly the same, very condescending.

Maybe I am funny about this because I am sure that many people don't get it, that I am against this.

Anyway that is why I am excited to have all these options nowadays because of Gavon and Mysticum.

Best regards

Hi Nick,

Slowing down a hardware gives you the chance to face new engines against old engines which is quite interesting and it is one of the features of a dream machine but slowing down by software means it is not bad when an emulator comes up. The emulator inside Mysticum gives you the chance to run old engines at their original speed. If you played MM IV, MMV, Rebel 5, Amsterdam, Roma or Dallas you would be playing against the same dedicated chess machines and you will have moves over the board from a strength of 1800 to 2100 depending of the machine you are playing. So that will also be quite interesting. One thing is true and it is also in your post, there are not much variety in programs that has been emulated. So here is when a dedicated machine can manage UCI engines makes them great.

Software and Hardware regards
Ricardo

[spacious_mind]

Just been reading up on the PXA320 which is used by Revelation II. It looks as if with it's 800 MHz has about 1000 MIPS. Which would mean that it is around 40% faster than Revelation 1. Based on this the ELO difference between Rev 1 and Rev 2 for say Hiarcs 13.3 would be around 28-30 points.

Perhaps someone could confirm.

Regards

Nick