The textbook feels heavier than the pencil because it has a greater mass and therefore exerts a larger force on your hand when you hold it. The weight of an object is determined by its mass and the force of gravity acting on it, so the textbook, being larger and denser than the pencil, feels heavier when you compare the two objects.
