o POSSIBLE SOLUTION(S)

+
The general case suggests that we make a square cut of length

x = a in each corner but that we extend the edge back from the perpendicular cut by a distance y = b so that we do not have an edge which is perpendicular to the base when we fold the edges, but rather a more open box in which the top cross section is larger than the bottom cross section.

+
If we do this we can see that we shall need the actual height of the box when we fold the edges. This height (h) is one leg of a right triangle in which the other leg is b and the hypoteneuse is a, hence by Pythagorean Theorem we have h = Sqrt[a^2 - b^2].

+
Thus now we can "see" that our folded up volume consists of four kinds of pieces:

+
(1) the inner box of dimensions (8 - 2a), (10 - 2a), and h,

(2) two prisms of length 8 - 2a and triangle cross sections with legs

b and h,

(3) two prisms of length 10 - 2a and triangle cross sections with legs b and h, and

(4) four quarters of a pyramid each with square base b by b and height h.

+
We add up these volumes to get the total volume of our box.

+
We offer a contour plot over the entire feasible region to help locate potential maxima.

+
Thus we go after a maximum near a = 2 and y = b. We set the partial derivatives with respect to a and b equal to zero.

+
Thus, ae, the square cut edge is 2.04789 cm.

+
And, be, the back cut is 1.08581 cm.

+
The box then has a base of dimensions

+
and a top of dimensions

+
And hence the maximum volume is 61.2466 cm^3 as seen below.

+
We now draw a picture of the figure - presuming we lay the flat sheet on the x-y plane with one vertex at the origin and the 8 cm side along the x- and the 10 cm side along the y-axis. Pi' are the points in the flat sheet which we cut into and form the bottom vertices of the box and the Qi's are the resulting top vertices.

+
We examine the square cut case when we cut a square piece out of each corner

+
We set up the volume for a square cut of a by a from each of the corners.

+
And maximize this volume.

+
Thus we should cut in an edge of length 1.47247.

+
In this case we do not make a back cut.

+
And hence the maximum volume is 52.5138 cm^3 as seen below.

+
A comparision between the back cut method and the square cut method shows that the former gives greater volume.

+
We note that we indeed get a greater volume by permitting the back cut for the volume in that case is 61.2466 cm^3.

+
We now draw a picture of the figure resulting from the simple case - presuming we lay the flat sheet on the x-y plane with one vertex at the origin and the 8 cm side along the x- and the 10 cm side along the y-axis. Pi' are the points in the flat sheet which we cut into and form the bottom vertices of the box and the Qi's are the resulting top vertices.

+
And finally we draw both figures together to compare them.