3×x^2×y+y^2×5/8×x^2

You don't need a "×" symbol to show multiplication. Use brackets or * or standard algebraic notation

e.g. 3×x^2×y ---> 3(x^2)(y) or simply 3x^2 y

Furthermore, you will need brackets to eliminate any ambiguity

I read yours as:

3x^2 y + (5/8)x^2 y^2

You don't have like terms so that is it
if you want it factored ....

= x^2 y(3 + (5/8)y)
= x^2(24 + 5y)/8
or
(1/8)x^2(24+ 5y)