POSSIBLE SOLUTION(S)
The general case suggests that we make a square cut of length
x = a in each corner but that we extend the edge back from the perpendicular cut by a distance y = b so that we do not have an edge which is perpendicular to the base when we fold the edges, but rather a more open box in which the top cross section is larger than the bottom cross section.
If we do this we can see that we shall need the actual height of the box when we fold the edges. This height (h) is one leg of a right triangle in which the other leg is b and the hypoteneuse is a, hence by Pythagorean Theorem we have h = Sqrt[a^2 - b^2].
Thus now we can "see" that our folded up volume consists of four kinds of pieces:
(1) the inner box of dimensions (8 - 2a), (10 - 2a), and h,
(2) two prisms of length 8 - 2a and triangle cross sections with legs
b and h,
(3) two prisms of length 10 - 2a and triangle cross sections with legs b and h, and
(4) four quarters of a pyramid each with square base b by b and height h.
We examine the square cut case when we cut a square piece out of each corner
A comparision between the back cut method and the square cut method shows that the former gives greater volume.
We note that we indeed get a greater volume by permitting the back cut for the volume in that case is 61.2466 cm^3.