After Tony McMahon and Christopher Routis saw their spot-kicks saved by Minstermen 'keeper Scott Flinders, York substitute Josh Carson won the the shoot-out for the hosts with the decisive penalty.
James Hanson's dramatic last-gasp equaliser had earlier taken the tie into extra time after James Berrett's 85th minute free-kick had looked likely to give York the win in normal time.
Routis initially gave City the lead of the tie midway through the first half before Luke Summerfield's penalty for York levelled things up early into the second period.
Before the drama of Berrett and Hanson's late goals, Alan Sheehan fired over the bar with a penalty for the Bantams in the 70th minute.
City's penalty loss at York was their first failure in ten shoot-outs.