Two straight walls meet at an angle of π/4. A straight piece of fencing PQ of length 10 metres is used to create an enclosure APQ.

a) Show that the area of APQ is given by 50(√2)sin[(3π/4)-ϕ]

b)If the angle ϕ is varied, using differentiation or otherwise find the value of ϕ which maximizes the area of
triangle APQ

you haven't defined angle ϕ, but I assume it is the angle between PQ and one of the walls, say at point P. If so, then

since A+P+Q = π,
Q = π-A-P = π - π/4 - ϕ = 3π/4 - ϕ
the law of sines says that
sinAQ/sinϕ = 10/sinA = P/sinQ
and you know that the area of the triangle is
1/2 AQ*PQ*sinQ
Now you can get your formula for the area in terms of ϕ
I don't think it is the one you gave ...
post your work if you get stuck.