so you have a total length of 40.
you use part of that length to make a circle with radius r. this circle will have circumference 2πr, so the leftover amount with which to make the square is 40 - 2πr.
therefore, each side of the square is one-fourth of (40 - 2πr), or 10 - 0.5πr.
the area of the square is thus (10 - 0.5πr)^2, so the answer (e) follows.
by the way, i would not recommend number picking on this problem, as the formulas are terribly awkward - not to mention the fact that you actually don't have to do that much work to derive the formula. the hard part is getting past the conceptual barrier of realizing that you have to subtract the circle's circumference from 40 in order to find out the amount of wire left to make the square.