old rules (Base Set - Neo4): when you retreat, pay the retreat cost, then flip a coin. On heads, it retreats normally, but on tails you can't retreat (and you can't try it again). when you attack, flip a coin first before doing anything else. On heads you perform the attack normally, but on tails you do 20 damage to yourself and the attack fails. (Apply weakness and resistance for self damage: e.g. Mewtwo will hit himself for 40 since he's weak to psychic, while Aerodactyl will not do any damage for he's got resistance to fighting.)
new rules (E1 - now, a.k.a. DF): retreat normally (pay retreat costs) without flipping coins when confused. When you attack, flip a coin before doing anything else. On heads the attack goes normally, but on tails place 3 damage counters on your Active and the attack fails. (i.e. don't apply weakness and resistance.)