I wrote:
The problem is that "can we cover the bigger rectangle with small rectangles such that no two rectangles overlap and no gap opens up"
Is it a correct usage of such that? Could I use "so that"? How about "in a way that"?
Is open ups the correct verb choice for gap here?
Such that is a natural and appropriate term to use here. So that is also acceptable, and less formal. In a way that is not grammatical, though in such a way that is - and is essentially what such that means. You could also use in a way that means, though that comes across as much more informal.
"No gap opens up" will be understood, but is not idiomatic, and some might argue as to whether it is semantically correct. "There are no gaps" would be more natural.
You also need to lose the that in "The problem is that". That quotation marks have you covered. If you didn't have the quotation marks, you'd want something like "the problem is to figure out if we can cover the bigger rectangle..."