As long as you can recognize a human face in the image, you're fine.
If you don't recognize who the particular face belongs to, recall that in Problem Set 2, after correctly implementing your Caesar cipher, there was a URL that could be decoded. If you don't remember, here is the encrypted link from the Problem Set 2 HTML form.
I have the same problem. After removing the 0xff red pixels I tried changing all remaining colored (all non-blank) pixels to black but that was definitely worse. Maybe that was going overboard. My next idea is to double the intensity of any non-whites.