RGB-based semantic segmentation has become a mainstream approach for visual perception and iswidely applied in a variety of downstream tasks. However, existing methods typically rely on highresolution RGB inputs, which may expose sensitive visual content in privacy-critical environments.Ultra-low-resolution RGB sensing suppresses sensitive information directly during image acquisition,making it an attractive privacy-preserving alternative. Nevertheless, recovering semantic segmentationfrom ultra-low-resolution RGB inputs remains highly challenging due to severe visual degradation.In this work, we introduce a novel fully joint-learning framework to mitigate the optimization conflicts exacerbated by visual degradation for ultra-low-resolution semantic segmentation. Experimentsdemonstrate that our method outperforms representative baselines in semantic segmentation performance and our ultra-low-resolution RGB input achieves a favorable trade-off between privacypreservation and semantic segmentation performance. We deploy our privacy-preserving semanticsegmentation method in a real-world robotic object-goal navigation task, demonstrating successfuldownstream task execution even under severe visual degradation.