To be honest, there's a lot we don't know about him. I think he's a very interesting character because we don't truly know why he left on Jim's birthday, and from a source Toby heard from, he left because he was having an affair with another woman, and it wasn't from James Sr. himself. Perhaps it's someone who knows him? Perhaps that "someone" was lying because they didn't want Toby involved to keep him safe, or perhaps James Sr. lied about the affair and left because of an even greater purpose we currently have no clue about.
Also, considering that there's a possible connection between the Lake family on Jim's father's side to Nimue, the Lady of the Lake, I find the supposed reason for him just walking out to be very odd. I hope he makes an appearance in Rise of the Titans, or at least we learn more about why he left in the first place.