Chasing South Africa's 165 for seven, the Black Caps measured their reply expertly for 15 overs to leave 25 for victory with 30 balls left.
But the Proteas pegged them back and left them somehow needing seven runs from the final over with five wickets in hand to overhaul South Africa's total of 165 for seven.
And they failed spectacularly to finish on 162 for seven in one of their most embarrassing international defeats as Marchant de Lange took two wickets in a final over that conceded only three runs.
Even a no-ball by De Lange from the last delivery of the innings could not save New Zealand.
When he overstepped, the New Zealanders needed a four from the last ball to win the match and series. But Tim Southee, newly at the crease and facing his first ball, played and missed outside off stump.
"When the pressure came on we were found wanting," Brendon McCullum, the New Zealand captain, said.
"Credit to South Africa, they kept the pressure on us and we didn't respond. Obviously we foundered at the end.
"That's not the way we want to play cricket.
"In pressure situations we want to play ruthlessly and to put other teams away, but we didn't do that tonight."