Adversarial policies in Go

Positional adversarial training

In December 2022, KataGo's main training run began to incorporate adversarial training using hand-written positions derived from cyclic attacks. We target the strongest KataGo network checkpoint from December 2023, kata1-b18c384nbt-s8526915840-d3929217702, which we dub dec23-victim. Our analysis identified two styles of attack: a fine-tuned variant of our original cyclic attack, and a qualitatively distinct "gift" attack that inexplicably leads the victim to gift the adversary two stones.

Gift attack

We discovered a new non-cyclic attack, which we call the "gift attack", that defeats dec23-victim at 512 visits of search in 91% of games. In this attack, the adversary sets up a "sending-two-receiving-one" situation where, for no valid reason, the victim gifts the adversary two stones and then needs to capture one back. However, the victim's recapture is blocked by positional superko rules. The adversary sets up the position to have the resurrection of one of its dead groups at stake, leading to a disaster for the victim. Although this attack does not scale up to high visits, it reveals additional unforeseen vulnerabilities in KataGo besides cyclic attacks. Explore randomly sampled games below.

In the first game, for example, we see a typical game unfold for the first 100+ moves. The victim takes an early lead, reaching 98% internal win probability by move 24, so the adversary is playing many moves that would normally be considered subpar. There is otherwise little overt evidence of anything suspicious happening at this stage. Gradually, we see the adversary playing some incursions into the victim’s territory, notably at moves 117 and 127. With correct play, these and nearby stones will eventually be captured.

At move 206, we see the first instance of the victim offering two stones, though in this position it is not yet dangerous. This happens repeatedly over the following moves, both at this location and on the left side. After move 263 though, when the adversary forms an eye at the top, any further gifts will lead to the victim inadvertently resurrecting the adversary’s entire group. This happens at move 344. After the adversary captures two stones at 345, the victim would like to recapture one stone but is barred by the positional superko rule. Thus, the adversary secures two eyes with move 347, saving its group that should have died. This is a significant reversal, though the victim still appears to be winning. But then, at move 366, the same scenario unfolds on the left side, tipping the scales decisively in favor of the adversary.

Human replication of gift attack

A Go expert (Kellin Pelrine) was able to learn and apply the gift attack to beat KataGo. This confirms a variety of attacks, and not just the original cyclic attack, can be replicated by humans. In the game below, Kellin (white) takes control of the left and bottom sides, while the KataGo victim (black) takes the top and right. It was relatively straightforward for Kellin to take control of the left and bottom since KataGo is in the lead due to controlling more of the center. Because of this, KataGo does not fight too hard for more territory, though Kellin still had to make sure the score difference would not get too extreme (some prior attempts failed due to insufficient points, even after receiving a "gift").

Beginning on move 70, Kellin (white) starts to set up a position to receive a gift. In moves 100 through 142, white pushes further into black’s territory, making sure that the gift will result in a very large swing in points. Then starting on move 154, white sets up another place for a gift. Black offers a gift on move 177, but white does not accept yet because black threatens to capture four stones on move 179. Note that this is the only place where black has a threat; white is careful to play very solidly and minimize potential threats, to be able to capitalize on the gift (noting that if there are lots of threats, the victim can use them to prevent a repeated board position and save themselves from the positional superko rule).

White finally sets up a third potential place for a gift starting on move 186. This was intended to maximize the chances. Normally all three of these groups would be dead, but on move 193, we see one instance of the attack succeed, with black sending two stones for no benefit and letting white resurrect their group. This happens again, with white’s largest “dead” group, on move 215. This completes the reversal, giving white a large lead, and black resigns shortly after.

We note, however, that this was completed at 1 victim visit. With more search, the attack seems much harder for humans (as well as our AI adversary); several human attempts at 256 and 512 visits failed. In particular, there was no issue establishing positions ready to receive a gift, as well as a stable gamestate on the rest of the board, but no gift was offered. We hypothesize that in addition to the key visible components of the attack (minimizing victim score lead, adversary positions that don’t have many threats against them, and setting up the shapes to receive gifts), there is a more opaque component of balancing value of moves across the board in the victim’s perception, such that it does not see much more valuable moves and offers the gift, but also does not search so much in the local area to see the disaster about to happen afterwards. This requires increasing precision at higher visits, and is difficult for humans to learn.

Continuous cyclic attack

We also fine-tuned a cyclic adversary to defeat dec23-victim at 4096 visits of search with a 65% win rate and at 65536 visits with a 27% win rate, showing that KataGo's adversarial training is insufficient to defend against cyclic attacks. Explore randomly sampled games from the adversary continuous-adversary below.

For example, in the first game, the adversary stakes out a sizable group in the center, around which the victim will form the cyclic group. We can see the outline of this group formed around move 65, and the victim completes a loose encirclement—the eventual cyclic group—around move 98. At move 122, the inside shape is completed with a double cut formation. This inside shape is distinctive of this adversary. In subsequent moves, the victim completes the cyclic group and the adversary begins to surround it. Move 210 is the last chance for the victim to escape (for example, by connecting where the adversary plays 211). After that the cyclic group is doomed, culminating in its final capture on move 217.

Human replication of continuous cyclic attack

A Go expert (Kellin Pelrine) studied this attack and was able to learn and implement it to beat KataGo. In the game (shown below), there were a number of key components to doing this successfully. The operation begins with the opening pattern with 3-3 points on moves 3 and 9, inducing KataGo (white) to play on the sides through move 18, which sets the stage to build the distinctive inside group shape. That operation begins in earnest with the cap at move 19. Move 27 is an abnormal move designed to induce 28 immediately and 36 later; an extension like this can be seen in some of our AI adversary's games. 41 induces 42, and through 47 the initial outline of the group is completed.

Move 59 begins the second phase, to induce the victim to create an eye for its cyclic group. 69 pushes white to capture a stone to make this happen, and 71 invites 72 which avoids the possibility of the victim making a false eye. The eye is finally completed when white captures at 108.

In the meantime, Kellin (black) begins to surround the cyclic group and invite connections to complete it. At the same time, with 149 and 173, black separates what will be an extension of the cyclic group from the rest of white's stones. With the connection at 232, the cycle itself is virtually complete. However, the inside shape does not yet have the distinctive double cut pattern. So after playing more moves to surround the cyclic group through 249, black buys some time with moves like 251 and 259, waiting for white to attack the inside group. This happens with 260, and the double cut is completed with 266.

At this point, the last phase begins. In a number of failed attempts, all the steps so far were insufficient to lead to a successful capture without the victim seeing the danger. To divert the victim's attention, black sets up a situation where the last chance to save the cyclic group will also be doing something else—specifically, capturing its tail. Black plays some preparatory moves at 273 and 275, which ensure black will not run into a liberty shortage with his own group. Finally, at move 277, black threatens to cut off white's tail, while at the same time filling liberties of the cyclic group. If white connects, the entire cyclic group will die. KataGo (white) seems to recognize that connecting will lead to disaster, and plays elsewhere, letting black capture with 279. Now, white has a final chance to save the cyclic group by capturing part of the inside group. Although capturing the tail is a huge gain for black, white still has a considerable lead if the cyclic group survives. However, white plays away at 280, letting black fill another liberty with 281. Now white can no longer capture the inside group before black captures the cyclic group, so white loses.

This attack was a lot more challenging to execute than the others on this page. We hypothesize that it is key to make the last threat before deciding the fate of the cyclic group look on the surface unthreatening to it. For example, in this case, it was a very valuable capture, a natural move for black to play even without any designs on the cyclic group. An inhuman weakness may have a surprisingly human component to it.

Big cyclic attack

After we disclosed the attacks of the previous two adversaries, KataGo added positions to its adversarial training dataset to defend against the attacks. But yet again we were able to fine-tune a cyclic adversary to beat a new KataGo model, may24-victim, at 65536 visits of search with a 56% win rate. may24-victim is the strongest KataGo model of the same size as dec23-victim as of August 2024.

A simple qualitative trait stands out with this adversary: it makes a bigger cyclic group. For example, in the first game below, we see through move 23 that its start to creating the cyclic group is wide and expansively open towards the center. It starts to outline the top boundary of the soon-to-be inside group with move 41, and the group is fleshed out through move 63. Over the next 50+ moves, the adversary takes a patient approach to playing around the fledgling cyclic group, intermittently scattering moves there in between moves on the right side. Around move 129, though, the adversary does start to surround the group in earnest. Notably, by connecting some of its stones together with moves like 179, the adversary ensures the surrounding groups have a decent number of liberties. This may be important to ensure the victim would have to think a bit further ahead to see disaster coming before the last opportunity to avert it (this concept has been observed with previous adversaries and is discussed in Appendix H of our first paper. On move 198, the victim completes the cycle. In this particular game, this move is also the one that locks in the disaster. After that, the victim makes some last ditch efforts to capture some surrounding stones, but the cyclic group is inevitably captured on move 233 and the game is decided.

Human replication of big cyclic attack

Go expert Kellin studied the big cyclic attack and was able to apply it to beat the dec23-victim at a higher visit count than before, winning at 4096 victim visits rather than 512. We attacked dec23-victim here instead of may24-victim since we used dec23-victim in previous human replications on this page and we wanted to compare the difficulty of executing the attack. We were able to beat dec23-victim at a higher number of visits than before.

In this game, to maximize chances of success, Kellin aimed for three things: a large cyclic group, many liberties on the surrounding groups ("moves-to-capture"; see Appendix H of our first paper), and an eye for the cyclic group. We see in the AI adversary's game records that the latter is not necessary, and moves-to-capture does not need to be that high either, but both these traits have been observed to help some past attacks, so were reasoned to potentially increase the odds here. The successful attempt here followed only two unsuccessful ones, suggesting it would be viable for humans with even higher visits. This is also far less than the replication of the continuous cyclic attack (which took over 20 initial attempts), indicating that for humans it is at minimum much easier to learn and most likely easier to execute too.

Kellin started inducing the cyclic group immediately with move 3. Move 21 is a classic tesuji to gain liberties, for what will become one of the surrounding groups. With moves like the sequence through move 31, move 65, and move 71, the key large inside group is formed, and correspondingly by move 106 the victim has outlined a large cyclic group. Along the way, when the victim played move 58, a place for the cyclic group to form an eye was established. With move 109 Kellin started to surround the cyclic group from above, and then on the right with move 141. On move 142, the victim seemed to catch a whiff of danger and formed an eye, which is a bit of an odd move given there are points remaining to be claimed in the top right and also many places where the cyclic group could run away. With move 172 and move 180, the cyclic group threatens to do so, creating a tense moment. But through move 191 the victim lets itself be walled off. After this, what's left is to fill in liberties and capture the cyclic group before the victim fully catches on and captures one of the surrounding groups. The deciding move is move 232, where if the victim had played against black's top left group, it could have forced a local stalemate (seki) between that group and its cyclic one, and secured infinite time to capture one of the other surrounding groups. But in the game, move 232 gives Kellin time to fill another liberty of the cyclic group, guaranteeing the capture. The victim makes a couple futile threats against the inside group, but the capture is completed on move 253, and the victim resigns.