A new Ratings list

Post by **Harvey Williamson** » Sat Mar 29, 2008 12:03 am

http://www.hiarcs.com/hiarcs_games.htm

Dylan Sharp · Post by **Dylan Sharp** » Sat Mar 29, 2008 1:29 am

Deep HIARCS 12 v Zappa Mexico UCI 63.5-36.5 (+42-15=43)
HIARCS 11.2 MP v Zappa Mexico UCI 59.5-40.5 (+34-15=51)

Zappa Mexico under-performs under these quick time controls.

I think a new rating list for reference is a good idea, though I'm not fully convinced of Chessbase book learning (if it's the one that plays the most played moves, not necessarily the best ones) but I'm looking forward for the results.

Hiarcs 12 is doing very good!

turbojuice1122 · Post by **turbojuice1122** » Sat Mar 29, 2008 1:54 am

The average of these first five tests gives Hiarcs 12 MP a rating of 70 elo higher than Hiarcs 11.2. Not bad...

fallstalk · Post by **fallstalk** » Sat Mar 29, 2008 2:15 am

what does this mean?

Deep HIARCS 12 v Rybka 2.3.2a UCI 51-49 (+32-30=28)

Dylan Sharp · Post by **Dylan Sharp** » Sat Mar 29, 2008 2:23 am

fallstalk wrote: what does this mean? Deep HIARCS 12 v Rybka 2.3.2a UCI 51-49 (+32-30=28)

Hiarcs 12 won 32 games, lost 30 and tied 28 for a total of 51 points over Rybka's 49.

turbojuice1122 · Post by **turbojuice1122** » Sat Mar 29, 2008 4:29 am

In the Rybka match, was Rybka in 64-bit and using the same number of cores as Hiarcs?

EDIT: Sorry, I should have read the match conditions first: "both processors used, 32-bit".

Vinvin · Post by **Vinvin** » Sat Mar 29, 2008 12:35 pm

To the hiarcs team : Could you update the chart here please : http://www.hiarcs.com/hiarcs_info.htm

Thanks !
Vincent

Krazyken · Post by **Krazyken** » Sun Mar 30, 2008 10:30 am

This at least gives an indication of the relative strengths of these programs and there books when they are playing at full strength.

I've been wanting to see such a list for quite some time. (but that should be their books)

Ray · Post by **Ray** » Mon Mar 31, 2008 8:46 pm

Surprisingly, there are no rating lists which test the complete chess program (engine+book) at blitz or rapid time controls. The SSDF and Selective Search rating lists do test this at tournament time controls of 3 minutes a move but as a result these lists take some considerable time to play enough games to rate a program.

Yes you're right, there is a gap there and it would be good to see it filled. I've been intending to run own book matches myself as an exercise on the side.

The other popular internet rating lists of the CEGT and CCRL do not allow engines to use their opening books and disable any position learning capabilities which handicaps programs like HIARCS which have learning features for positions and their books.

I don't see how you can get a reliable ratings list with learning turned on. For example, did Hiarcs play Rybka as it's first match with no accumulated learning, or did it play Rybka last and then have the benefit of accumulated learing at that point, and therfore get a better result ? The sequence in which you play games could affect the rating significantly, couldn't it ? Please correct me if I'm wrong, it's a question not a criticism

It's a pity that you're not using a 64-bit O/S, because programs like Rybka are severely handicapped on a 32-bit platform. Still, you can argue that it is a common platform, equal for all, and you'd be right, but it's a platform that will ensure Hiarcs gets a better result comparatively by not letting the 64-bit engines show their prowess. Fork out £50 for XP X64 and start again

Still good to see though.

Post by **Harvey Williamson** » Mon Mar 31, 2008 8:52 pm

I think very few customers have a 64bit OS and it will stay that way for a long time. I think many now have a dual core system but very few other than us Computer Chess geeks are running a 64bit os - so for the majority of customers 32bit results are more interesting. Surely the main purpose of any ratings list is to show what you can expect on the average hardware of most customers not the geeks.

Ray · Post by **Ray** » Mon Mar 31, 2008 9:19 pm

I would think that most of your customer base are chess geeks, and most of them have 64-bit O/S. But who knows - you presumably don't have the information either as to what your customers have. So it's an unknown. But one thing is for certain, you won't see Rybka competing in major tournaments with 32-bit

Any comment on the learning issue I raised ?

Post by **Harvey Williamson** » Mon Mar 31, 2008 9:24 pm

Ray wrote:I would think that most of your customer base are chess geeks, and most of them have 64-bit O/S. But who knows - you presumably don't have the information either as to what your customers have. So it's an unknown. But one thing is for certain, you won't see Rybka competing in major tournaments with 32-bit

Any comment on the learning issue I raised ?

Your wrong about 64bit very few are interested in it, at the moment. If an author like Mark or Stefan spend time working on learning it should be used - if the other authors dont implement it it is their problem. If you load the engine at home learning is on - if you load it on playchess learning is on.

So your ratings list is based on rybka in tournaments? That wont help a customer at home decide with his 1 or 2 cpu 32 bit machine.

Ray · Post by **Ray** » Mon Mar 31, 2008 9:43 pm

It's your list and you choose the conditions for it - no problem with that at all. It's very useful 32-bit or 64-bit and a welcome addition.

I don't think any list out there can predict tournament performance and none claim to. Too many variables in those - different hardware, bigger hardware, special books, it makes them exciting and interesting and nail-biting. Paderborn went down to the wire and what a result !!

I really only wanted a comment on this, to increase my understanding, which you haven't given yet

I don't see how you can get a reliable ratings list with learning turned on. For example, did Hiarcs play Rybka as it's first match with no accumulated learning, or did it play Rybka last and then have the benefit of accumulated learning at that point, and therfore get a better result ? The sequence in which you play games could affect the rating significantly, couldn't it ? Please correct me if I'm wrong, it's a question not a criticism

Post by **Mark Uniacke** » Tue Apr 01, 2008 2:54 am

Hi Ray,

The SSDF have been testing with learning for 15 years and I certainly would NOT question the reliability of their rating list - I think it is very reliable.

Graham Banks · Post by **Graham Banks** » Tue Apr 01, 2008 3:36 am

Ray has a good point though Mark.

Even if a rating list is constructed by one sole tester, engines accumulate learning at different rates.
When a new engine gets added to such a rating list, then it plays with no learning as yet against engines which could have months or years of accumulated learning.

Then if you have more than one tester involved in constructing such a list, the learning accumulated by an engine on one tester's computer is different to the learning accumulated by the same engine on the computers of other testers.
Testing with learning off circumvents this problem.

It's a tricky issue.

Regards, Graham.

PS - thanks for Hiarcs 12 and keep up your efforts.

HIARCS Chess Forums

A new Ratings list

A new Ratings list

Re: A new Ratings list

crumbs from the table

Re: crumbs from the table