If a thought pops into your head, and you choose not to follow that train of thought, then it was not your conscious self that brought it on. It was more likely something else that was stored away, or a neuro-chemical trigger. What you choose to do next is your conscious response.
When we put information in, our brain builds connections to related thoughts, so it would be good to think of ways to break the two topics from being associated. That is, consciously lead your brain into a different direction, to set up a new pattern.
I find memories pop up when a smell connects two unrelated topics. Like when I listen to a radio interview while painting a room, the smell will later remind me of the interview.
That's a good reason for keeping everything pure... when priests used to enter the inner courts they had to do ritual cleansing and change their robes. There were no distractions. These days, a person can switch browser tabs from online scriptures to work mail to gorilla behavior research to windmill debates... so the brain could easily link together this and that.
There are medical conditions where people find this a predictable symptom, so in a case like that you would know it's not your fault. Just keep aiming toward staying in positive communications with your Heavenly Father.