Depends where you are and what depth of experience you want. If you're out in the countryside or off the well-beaten tourist path, then yes, you need some Japanese to do more than wander about. In the big cities and major tourist spots, though, they either have ample English signage or are used to dealing with foreigners who don't speak Japanese wandering about. People with no Japanese ability regularly come to the country and have a great time -- I've known many, and been one myself.
Would it be a richer experience if you could read all the signs and say more than a few survival phrases? Without question. Do you need to hit N3 on the Japanese Language Proficiency Test before even considering a visit? Not in the slightest.