It can be very tricky to remove glue from rubber sometimes. I don’t think really there’s any chemicals that will help to much with this. I don’t think it’s a good idea to use acetone as it will probably damage the rubber. To be honest the best way is to just take your time using your thumb and rub the glue off slowly. It’s a long process but this way you won’t damage the sponge underneath.