Some food for thought

This forum is for general discussions and questions, including Collectors Corner and anything to do with Computer chess.

Moderators: Harvey Williamson, Steve B, Watchman

Forum rules
This textbox is used to restore diagrams posted with the fen tag before the upgrade.
Post Reply
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Some food for thought

Post by spacious_mind »

I am quite in awe at the speed I am getting with my new desktop pc (which I am sure is not the fastest in the world) without trying to overclock my computer.

It seems that I can get in excess of 35 Billion positions per hour.

To put that in perspective. Best dedicated computers averaged at around 8000 positions per second.

Therefore for a 50 move Tournament level game (40m in 2 hrs) with ponder on. A top dedicated would calculate about:

18,000 seconds x 8,000 = 144,000,000 Total nodes.

My modern computer running Stockfish achieves:

10,000,429 nodes per second therefore equivalently would play the whole of the old top dedicated computer 50 move match with ponder in 144,000,000 / 10,000,429 = 14.4 seconds

Therefore my PC is approx. 12,500 times faster than the old top dedicated computer.

OK since I would not want the PC to ponder I would have to turn Ponder off and set the PC up to play 50 moves in 28.8 seconds against the dedicated setting of Ponder On and 3 minutes per move.

Or with both playing Ponder Off the setting would have to be:

PC = 50 moves in 14.4 seconds (ponder off)
Dedicated = 3 minutes per moves (ponder off)

Calculation at 60 ELO gain per speed doubling
2x = +60 ELO gain
4x = +120 ELO gain
8x = +180 ELO gain
16x = +240 ELO gain
32x = +300 ELO gain
64x = +360 ELO gain
128x = +420 ELO gain
256x = +480 ELO gain
512x = +540 ELO gain
1024x = +600 ELO gain
2048x = +660 ELO gain
4092x = +720 ELO gain
8188x = +780 ELO gain
12500x = +810 ELO gain approx.

Now take Stockfish 4 rated at 40/40 at ELO3220 and remove 810 ELO you get to ELO 2410 when setting it up to a comparable setting to a top dedicated computer.

So in summary (If that were possible). If I take Stockfish 4 or any other top engine and set it up at around 28.8 seconds per 50 moves with ponder off (or 30 seconds per 50 moves) and play it against a top dedicated computer playing at 3 minutes per move and ponder on, I should get some highly interesting and competitive games :P

ps. would probably have to clear hash as well for the PC after every move since otherwise this would also become an unequal advantage.

So setting would have to be something like:

PC = 50 moves in 30 seconds Ponder Off and Clear Hash after every move versus Dedicated = 40 moves in 2 Hours with Ponder on.

Amazed regards
Nick
Wardy
Full Member
Posts: 940
Joined: Tue Jul 31, 2007 7:17 pm
Location: Wellingborough

Post by Wardy »

If the dedicated and pc engine use brute for search then there would be a correlation along what you suggest ( assuming the evaluation overhead was similar), however none but the very oldest kit/software would do so ( I'm thinking White Knight on the BBC Model B)

I'm not sure how similar different the selective searches work across different software/hardware...........I suspect quite a bit so the correlation would perhaps be different.

Given advances in software techniques I suspect the dedicated would need a larger time advantage
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Post by spacious_mind »

Wardy wrote:If the dedicated and pc engine use brute for search then there would be a correlation along what you suggest ( assuming the evaluation overhead was similar), however none but the very oldest kit/software would do so ( I'm thinking White Knight on the BBC Model B)

I'm not sure how similar different the selective searches work across different software/hardware...........I suspect quite a bit so the correlation would perhaps be different.

Given advances in software techniques I suspect the dedicated would need a larger time advantage
Hi Wardy,

It would be a good test I am sure. Might do it someday. I did a similar 5 ply test a while ago and the modern engines struggled against the dedicated computers who also played under their selective settings.

Also remember that the top dedicated had their selective search as well. It is a matter of knowing which selective was better. Selective then or Selective now.

MY thoughts here are let them play under their best settings and just manipulate the time given per move. Problem is you cannot do that with engines unless you turn ponder off and also engines store game evaluation in memory so hash would have to be reset, but it would be worth trying it with hash and with hash reset on every move to see what the effect is.

ps. with Hash on you might have to revert to the 14.4 seconds per 50 moves for the engine (or 15 seconds per 50 moves).

another ps. also anyone trying this would have to do their own time calculation based on their own PC hardware.

Regards
Nick
Nick
Wardy
Full Member
Posts: 940
Joined: Tue Jul 31, 2007 7:17 pm
Location: Wellingborough

Post by Wardy »

It's definitely interesting for sure, I suspect that the selective search methods have improved by a factor that makes it significant.

I think it would take a lot of experimentation to get to the point where stockfish runs at a similar elo to say a Tasc R30 using direct games :-)

Should the starting point be engines with adjustable elo to get a good match between a pc and a dedicated.........set Hiarcs to 2400 for example?
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Post by spacious_mind »

Wardy wrote:It's definitely interesting for sure, I suspect that the selective search methods have improved by a factor that makes it significant.

I think it would take a lot of experimentation to get to the point where stockfish runs at a similar elo to say a Tasc R30 using direct games :-)

Should the starting point be engines with adjustable elo to get a good match between a pc and a dedicated.........set Hiarcs to 2400 for example?
Hi Wardy,

Setting engines to an ELO restriction has been tried and tested against dedicated computers. The games are artificial and kind of boring. You have to have conditions where both are playing at their best.

The test is play the R30 against Stockfish or a TM Lang against engine under the conditions I just mentioned. I think we might be surprised.

Regards
Nick
User avatar
Theo
Member
Posts: 132
Joined: Tue Mar 05, 2013 11:34 am

Post by Theo »

Hi Nick,

I dont know about Stockfish, but some engines like Deep Rybka offer CPU usage settings, e.g. you can set Rybka to use only 1% CPU time. Maybe that would ease things a bit. Other possibility would be to use a external CPU time limiter like the free "Battle Encoder Shirase".

Using CPU speed limitation for the engine would allow the dedicated opponent to ponder at least more than 30 secs per game.

Kind Regards,
Theo
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Post by spacious_mind »

Theo wrote:Hi Nick,

I dont know about Stockfish, but some engines like Deep Rybka offer CPU usage settings, e.g. you can set Rybka to use only 1% CPU time. Maybe that would ease things a bit. Other possibility would be to use a external CPU time limiter like the free "Battle Encoder Shirase".

Using CPU speed limitation for the engine would allow the dedicated opponent to ponder at least more than 30 secs per game.

Kind Regards,
Theo
Hi Theo,

Yes I know, that concept is what I use in Mysticum very successfully and easily, I can get an engine to play at the equivalent speed of a 68020-12 Mhz and I do that all the time.

But here the topic is doing this with a high end PC and no throttling, just level settings.

Regards
Nick
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Post by spacious_mind »

OK the Test is on.

[fen]r1bq1rk1/ppp2ppp/2np1b2/1B1N4/3pP3/5N2/PPP2PPP/R2QR1K1 w - - 0 10[/fen]

After 9 moves:

[Event "Test Match - Dedicated against Engine"]
[Site "Pelham, AL, USA"]
[Date "????.??.??"]
[Round "?"]
[White "Tasc CM32 512K King 2.2, 180S."]
[Black "Stockfish 4, 100/1M."]
[Result "*"]
[WhiteElo "2400"]
[BlackElo "2410"]
[PlyCount "19"]
[EventDate "2013.11.11"]
[EventType "match"]
[EventCountry "USA"]

1. e4 e5 2. Nf3 Nc6 3. Nc3 Nf6 4. d4 {Stockfish 4 out of book} exd4 5. Nd5 Bc5
{Tasc CM32 KIng 2.2 Agg out of book} 6. Bg5 Be7 7. Bxf6 Bxf6 8. Bb5 O-O 9. O-O
d6 10. Re1 *



I realized after I started that ponder is not really that effective since Stockfish moves instantly. Anyway will continue under this setting and see what happens. The Stockfish setting is 100 moves in 1 minute. Next game if needed I will do 200 moves in 1 minute.

Regards
Nick
Nick
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Post by spacious_mind »

[fen]r5rk/p4p1p/2p5/1pp2Q2/3bP2P/3B1R2/q4PP1/3R2K1 w - - 0 29[/fen]

[Event "Test Match - Dedicated against Engine"]
[Site "Pelham, AL, USA"]
[Date "????.??.??"]
[Round "?"]
[White "Tasc CM32 512K King 2.2, 180S."]
[Black "Stockfish 4, 100/1M."]
[Result "*"]
[WhiteElo "2400"]
[BlackElo "2410"]
[PlyCount "56"]
[EventDate "2013.11.11"]
[EventType "match"]
[EventCountry "USA"]

1. e4 e5 2. Nf3 Nc6 3. Nc3 Nf6 4. d4 {Stockfish 4 out of book} exd4 5. Nd5 Bc5
{Tasc CM32 KIng 2.2 Agg out of book} 6. Bg5 Be7 7. Bxf6 Bxf6 8. Bb5 O-O 9. O-O
d6 10. Re1 Be5 11. c3 Bg4 12. h3 Bxf3 13. Qxf3 dxc3 14. bxc3 Nb8 15. Qe3 g5 16.
Be2 c6 17. Nb4 Qf6 18. Red1 Bxc3 19. Rab1 Nd7 20. Rb3 Be5 21. Nd3 Bd4 22. Qg3
b5 23. h4 Nc5 24. Nxc5 dxc5 25. Rf3 Qe6 26. Qxg5+ Kh8 27. Bd3 Rg8 28. Qf5 Qxa2
*

Interesting position. This setting still gives Black about 4 times as many nodes as needed and goes to a depth of 22 ply against King 2.2 with 9 ply. Black has advantage but King has made a bit of game of it so far.

Definitely adjustments needed in game 2.

Regards
Nick
Wardy
Full Member
Posts: 940
Joined: Tue Jul 31, 2007 7:17 pm
Location: Wellingborough

Post by Wardy »

Interesting stuff, thanks for posting the results.

We are in fact comparing the whole search and evaluation code, it won't purely be a measure of nodes that brings us to an idea of what parity might be.

If you want to boost nodes the stick hyperthreading on and run on those virtual cores............. the node count will increase but the actual quality of analysis will likely drop as the extra virtual nodes count for nothing.......
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Post by spacious_mind »

Here is the final position:

[fen]r7/5k1p/2p5/4P3/2pbBP1P/pp4R1/4K1P1/8 w - - 0 40[/fen]

[Event "Test Match - Dedicated against Engine"]
[Site "Pelham, AL, USA"]
[Date "????.??.??"]
[Round "?"]
[White "Tasc CM32 512K King 2.2, 180S."]
[Black "Stockfish 4, 100/1M."]
[Result "0-1"]
[ECO "C47"]
[WhiteElo "2400"]
[BlackElo "2410"]
[Annotator ",Microsoft"]
[PlyCount "78"]
[EventDate "2013.11.11"]
[EventType "match"]
[EventCountry "USA"]

1. e4 e5 2. Nf3 Nc6 3. Nc3 Nf6 4. d4 {Stockfish 4 out of book} exd4 5. Nd5 Bc5
{Tasc CM32 KIng 2.2 Agg out of book} 6. Bg5 Be7 7. Bxf6 Bxf6 8. Bb5 O-O 9. O-O
d6 10. Re1 Be5 11. c3 Bg4 12. h3 Bxf3 13. Qxf3 dxc3 14. bxc3 Nb8 15. Qe3 g5 16.
Be2 c6 17. Nb4 Qf6 18. Red1 Bxc3 19. Rab1 Nd7 20. Rb3 Be5 21. Nd3 Bd4 22. Qg3
b5 23. h4 Nc5 24. Nxc5 dxc5 25. Rf3 Qe6 26. Qxg5+ Kh8 27. Bd3 Rg8 28. Qf5 Qxa2
29. Qxf7 Qxf7 30. Rxf7 Rg7 31. Rxg7 Kxg7 32. Kf1 a5 33. Ke2 a4 34. Bb1 b4 35.
Rd3 b3 36. Rg3+ Kf6 37. f4 c4 38. e5+ Kf7 39. Be4 a3 0-1

In the end this was quite an easy win for Stockfish. Tournament setting of 100 moves in 1 minute does not really work for a couple of reasons.

1) Stockfish does not stick to the move rules in Arena and therefore you cannot play this setting. 40 moves were played in 35 seconds.
2) The search depth is still much too deep. At least 4 times then what it should be. So even if this setting worked then you would have to do something like 400 moves in 1 minute.

So this is a bust with Arena. Maybe it works better under Winboard, but I doubt it.

So as a second test I tried to play instead by putting in the equivalent nodes. So for the first game I tried 240,000 nodes for Stockfish and King 2.2 playing at 30 seconds per move:

[Event "Test Match - Dedicated against Engine"]
[Site "Pelham, AL, USA"]
[Date "2013.11.11"]
[Round "?"]
[White "Stockfish 4, 240k/N"]
[Black "Tasc CM32 512K King 2.2, 30S."]
[Result "1-0"]
[ECO "E12"]
[WhiteElo "2410"]
[BlackElo "2400"]
[Annotator ",Microsoft"]
[PlyCount "51"]
[EventDate "2013.11.11"]
[EventType "match"]
[EventCountry "USA"]

1. d4 Nf6 2. c4 e6 3. Nf3 b6 4. Nc3 Bb7 5. a3 d5 6. cxd5 exd5 {
Stockfish 4 out of book} 7. Bf4 {Tasc CM32 512K King 2.2 out of book} Ne4 8.
Nxe4 dxe4 9. Ne5 f6 10. e3 fxe5 11. Qh5+ Kd7 12. dxe5 Bd5 13. e6+ Kc8 14. Rd1
c6 15. Bc4 a5 16. Bxd5 cxd5 17. O-O Kb7 18. Rxd5 Qf6 19. Rc1 Qxb2 20. Rc7+ Ka6
21. Qd1 Bc5 22. a4 Ra7 23. Qf1+ b5 24. Rcxc5 Rb7 25. Rxb5 Qxb5 26. Rxb5 1-0

This was also a bust Stockfish absolutely hammered King 2.2.

Setting was MPV = 1 Threads= 8

Again the search depth was too deep in comparison to old dedicated.. ie Stockfish about 13/14 ply deep and King 2.2 7 ply deep at 30 seconds.

That is just too far apart.

So for a 3rd attempt I tried to reduce the Nodes down to 60,000 and play King again at 30 seconds per move:

[Event "Test Match - Dedicated against Engine"]
[Site "Pelham, AL, USA"]
[Date "????.??.??"]
[Round "?"]
[White "Tasc CM32 512K King 2.2, 30S."]
[Black "Stockfish 4, 60k/N"]
[Result "1/2-1/2"]
[ECO "D27"]
[WhiteElo "2400"]
[BlackElo "2410"]
[Annotator ",Microsoft"]
[PlyCount "156"]
[EventDate "2013.11.11"]
[EventType "match"]
[EventCountry "USA"]

1. d4 d5 2. c4 dxc4 3. Nf3 e6 4. e3 Nf6 5. Bxc4 c5 6. O-O a6 7. dxc5 Bxc5 {
Tasc CM32 512K King 2.2 out of book} 8. Qxd8+ {Stockfish 4 out of book} Kxd8 9.
Nc3 Bd7 10. e4 b5 11. Bd3 b4 12. Ne2 Ke7 13. Bg5 h6 14. Bxf6+ gxf6 15. Rac1 Rc8
16. Ned4 Bb5 17. Nxb5 axb5 18. Ra1 Nc6 19. Rfc1 Ne5 20. Nxe5 fxe5 21. b3 Kd6
22. Rc2 Bd4 23. Rxc8 Rxc8 24. Rd1 Ra8 25. Rd2 Kc5 26. Rc2+ Bc3 27. g3 Kd4 28.
Bxb5 Kxe4 29. h4 Kd4 30. Bc6 Rc8 31. Bb5 e4 32. Re2 Ke5 33. Ba6 Rg8 34. Kg2 f5
35. Rc2 f4 36. a4 fxg3 37. fxg3 Be1 38. Re2 Bxg3 39. Bb7 Bxh4+ 40. Kh3 Be7 41.
a5 Bc5 42. Rxe4+ Kd6 43. Rg4 Rf8 44. Rg3 Rf7 45. a6 Rf4 46. Rd3+ Kc7 47. Rd2
Kb6 48. Ra2 Ka7 49. Kg2 e5 50. Ra5 Bd4 51. Rb5 Bc3 52. Rc5 Rf6 53. Rc8 Kb6 54.
Rc4 h5 55. Rh4 Rf5 56. Rc4 Be1 57. Rc6+ Ka7 58. Re6 Bh4 59. Kh3 Bf6 60. Bd5 Rf2
61. Bc4 Ka8 62. Rd6 Kb8 63. Rb6+ Ka8 64. Kg3 Rf5 65. Rxb4 h4+ 66. Kg4 Rf4+ 67.
Kh3 Rf3+ 68. Kg2 Rg3+ 69. Kf2 Rg7 70. Rb6 Bg5 71. Re6 Ka7 72. Bd5 Bf4 73. b4
Rd7 74. Bf3 Bg3+ 75. Ke2 e4 76. Bxe4 Rd6 77. Rxd6 Bxd6 78. b5 h3 1/2-1/2

This was a great game!! Stockfish 4 did blunder in this position:

[fen]8/k7/P3Rb2/4p2p/1pB5/1P5K/5r2/8 w - - 0 61[/fen]

Stockfish played 61. ... Ka8? here which perhaps should have lost after 62. Re8+!

But it was not to be and a draw was a fair result. This setting did allow for some exciting chess though!

The search depth is still about 12 to 13. So I am tempted to drop it down a notch further from 60,000 to 30,000 for 30 seconds per move dedicated opponents.

So as a standard it might be worth trying out the following computer nodes in Arena against dedicated computers:

30 Seconds per move = 30,000 Engine Nodes
1 Minute per move = 60,000 Engine Nodes
3 Minutes per move = 180,000 Engine Nodes.

Stockfish 4 is definitely capable of playing a great and competitive game against dedicated computers.

Nice thing about doing this with nodes is that the standard is the same for all PC. Hardware is no longer a factor.

ps... Node setting does not impede Stockfish in any way as it completes it's next closest upwards ply search in any case. Which in a sense gives it an average time feeling, albeit always on the higher side of average. So there is no cutting of half way in its search.

Regards
Nick
User avatar
spacious_mind
Senior Member
Posts: 3999
Joined: Wed Aug 01, 2007 10:20 pm
Location: Alabama
Contact:

Post by spacious_mind »

In this game I have dropped the nodes down to 30,000 for Stockfish for.

[Event "Test Match - Dedicated against Engine"]
[Site "Pelham, AL, USA"]
[Date "2013.11.11"]
[Round "?"]
[White "Stockfish 4, 30k/N"]
[Black "Tasc CM32 512K King 2.2, 30S."]
[Result "1-0"]
[ECO "B33"]
[WhiteElo "2410"]
[BlackElo "2400"]
[Annotator ",Microsoft"]
[PlyCount "129"]
[EventDate "2013.11.11"]
[EventType "match"]
[EventCountry "USA"]

1. e4 c5 2. Nf3 e6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 Nc6 6. Ndb5 d6 7. Bf4 e5 8.
Bg5 a6 {Stockfish 4 out of book} 9. Na3 Be6 10. Bxf6 {
Tasc CM32 512K King 2.2 out of book} Qxf6 11. Nd5 Bxd5 12. exd5 Ne7 13. c4
O-O-O 14. Be2 g6 15. Nb1 Kb8 16. O-O Bg7 17. Nc3 Qh4 18. g3 Qd4 19. Qc2 f5 20.
Rfd1 Qc5 21. Qd2 e4 22. Rab1 Rhe8 23. b4 Qb6 24. b5 e3 25. fxe3 Bh6 26. Kh1
Bxe3 27. Qd3 Nc8 28. c5 Qxc5 29. bxa6 b6 30. Rb5 Qc7 31. Rdb1 Qg7 32. Na4 Rd7
33. Qb3 Ka8 34. Rb4 Ree7 35. Bb5 Rd8 36. Bc6+ Ka7 37. Nxb6 Bxb6 38. Rxb6 Re1+
39. Rxe1 Nxb6 40. Qb4 Qf6 41. Kg2 Rb8 42. Re6 Qd8 43. Rxd6 Qc7 44. Re6 Nc8 45.
Qd4+ Rb6 46. Re8 f4 47. Bb7 Qc2+ 48. Kf3 Qxh2 49. Bxc8 Qxg3+ 50. Ke2 Qg2+ 51.
Kd3 Qf3+ 52. Kd2 Qg2+ 53. Kd1 Qf1+ 54. Re1 Qf3+ 55. Re2 g5 56. d6 g4 57. d7
Qf1+ 58. Kd2 Qb1 59. d8=Q Qxa2+ 60. Ke1 Qa5+ 61. Kf1 Qb5 62. Q4xb6+ Qxb6 63.
Re7+ Qc7 64. Qxc7+ Ka8 65. Qa7# 1-0

Stockfish 4 wins but the game was very competitive. King 2.2 on Aggressive setting is maybe too aggressive. Stockfish has an interesting game style.

This game was a lot of fun.

Regards
Nick
Post Reply