men in black - Why did J remember about K when no one else did?

In Men in Black 3, when J is about to do the time jump thing, he asks Jeffrey why he remembers about K while no one else does. Jeffrey says something like "Oh, so you were there", but it's never actually explained (or at least, I didn't get it) what that means, or why J is the only one that has his memories from the other timeline. So, why is that?