I think I found the full answer...
U-57
Claus Korth (Topp didn't take command until June 1940)
HMS Nelson
30th October 1939, near the Orkneys
All 3 torpedoes failed to detonate
And I just realized the attack occurred only 16 days after Prien sank the Royal Oak!
The spirit of the old Admiral himself must have been watching over HMS Nelson that day.
I think that sort of thing happened more than once. I recall reading an account of a U-boat commander who managed to fire off a spread against a battleship and all of them prematured. His presence was given away too soon and he had to endure a working over by the escorts. He reportedly was so ticked off over it that he had to be hospitalized. Don't know if that last bit is true or apochryphal, but I think I would need a visit to the loony bin after something like that.
hmmm that's interesting.... I have that incident down to U-56, OberLt Wilhelm Zahn, three hits on HMS Nelson, near Orkneys, 30 Oct 1939. Perhaps it needs more checking.
I'd been reading more about Zahn and found he'd made five patrols on U-56, sinking only one ship, a small Swedish (neutral), and damaging one small British ship. He also made four patrol on U-69 without any successes at all. The kaleun that relieved him on U-69 then took it out and sank six ships in two patrols, being lost on the third (U-69's 11th) patrol.