Can you put (grin) in a sentence where it acts as direct object?

If you don't apologize, I'll wipe that grin off your face.