I've had that before but only on grey grout strangely, but just used a very sharp stanley blade once it had gone off. It removed the very thin layer of Silicon and the grout was fine.
Never really goes I'm afraid. Scratch it off with your finger nails (honestly)
In future be really anal about clearing the joints with your profiling tool. I prefer to scrape any excess back in toward the bead (rather than cutting into it and scraping it away as that can add to the smearing) and then going over the bead again to make all neat again.