This should not be the case for everyone, but for me, this will be a lifelong problem, since I have no vestibular function on either side. Open spaces are always worse because I have less solid, unmoving objects to fix my gaze for stabilization as I am walking. I have to look down to make sure there are no uneven surfaces because if I lose my footing and begin to fall, I have very little ability to regain my balance. Therefore, I have learned to stay close to a wall, railing, or another person when I am outdoors, whenever possible. I can never go across an open space in the dark without assistance.
But, you know what, many people have to make modifications in lifestyle for many different health reasons and I have learned that it has made very little difference in the "grand scheme" of things.
However, to respond to your "maybe I should walk that way more often to challenge myself" ...... yes, you should, but I would always do it safely. A bad fall is never a good thing. Try walking up and down the hallway in your house, first turning your head from side to side and then again turning your head up and down. The goal is to walk as steadily as you can while increasing the speed with which you are moving your head. This exercise should carry out to walking in open spaces because you are strengthening your vision and sensorineural systems.
Best thoughts. Clarice