Going too close to Shadow Keep triggers the event . It also seems to trigger a little bit past the furnace golem if you take the Bonny Village route

Not even, it happened when I took like 5 steps past the grace just after the village

Getting too close to the shadow keep. You can be at its front door, just past the fire golem, or somewhere in the east after the Bonny route