Close and precise placement of irregularly shaped objects requires a skilled robotic system. The manipulation of objects that have sensitive top surfaces and a fixed set of neighbors is particularly challenging. To avoid damaging the surface, the robot has to grasp them from the side, and during placement, it has to maintain the spatial relations with adjacent objects, while considering the physical gripper extent. In this work, we propose a framework to learn an agent based on reinforcement learning that generates end-effector motions to place objects as close as possible next to each other. During the placement, our agent considers the spatial constraints with neighbors defined in a given layout of the objects while avoiding collisions. Our approach learns to place compact object assemblies without the need for predefined spacing between objects as required by traditional methods. We thoroughly evaluated our approach using a two-finger gripper mounted on a robotic arm with six degrees of freedom. The results show that our agent significantly outperforms two baseline approaches in object assembly compactness, thereby reducing the space required to place the objects according to the specified spatial constraints with the neighboring placed objects.