1. A set of instructions that 'tell' you how to cook something
[What is the antecedent of "that tell..."? Is it 'instructions' or 'set'?]
2. A set of instructions that "tells" you how to cook something
[Can we use 'tells' here?]
1 is wrong.
The singular word, "set," is the subject of "tells."