In cluttered scenes, and with unknown objects, typical pick and place pipelines are sub-optimal. Assembly and rearrangement tasks require flexible manipulation modes using both prehensile and non-prehensile approaches with multimodal perception.