Searching for objects in cluttered environmentsrequires selecting efficient viewpoints and manipulation actionsto remove occlusions and reduce uncertainty in object locations,shapes, and categories. In this work, we address the problemof manipulation-enhanced semantic mapping, where a robot hasto efficiently identify all objects in a cluttered shelf. AlthoughPartially Observable Markov Decision Processes (POMDPs) arestandard for decision-making under uncertainty, representingunstructured interactive worlds remains challenging in thisformalism. To tackle this, we define a POMDP whose belief issummarized by a metric-semantic grid map and propose a novelframework that uses neural networks to perform map-space be-lief updates to reason efficiently and simultaneously about objectgeometries, locations, categories, occlusions, and manipulationphysics. Further, to enable accurate information gain analysis, thelearned belief updates should maintain calibrated estimates of un-certainty. Therefore, we propose Calibrated Neural-AcceleratedBelief Updates (CNABUs) to learn a belief propagation modelthat generalizes to novel scenarios and provides confidence-calibrated predictions for unknown areas. Our experiments showthat our novel POMDP planner improves map completeness andaccuracy over existing methods in challenging simulations andsuccessfully transfers to real-world cluttered shelves in zero-shotfashion.