Following goals from James Hanson and the in-form Filipe Morais, Parkinson's men looked in control and set for the three points at Vale Park until Jordan Pickford was penalised for bringing down Chris Birchall inside the box late on.
Along with the stoppage time penalty award, Pickford was also given his marching orders.
Michael O'Connor stepped up to slot home the 94th minute spot-kick - cruelly robbing City of their seventh away win in the league in the process.
Parkinson said: "It was a really good performance from the lads today (Saturday) - it is just frustrating that we couldn't see the game out at the end there.
"Even after they scored, we have responded brilliantly and got the goal back almost straight away again.
"We had a couple of other really great chances to score in the second half. Not just good chances, gilt-edge ones, I don't know the goalkeeper has kept one of them out.
"Even though we didn't get that third goal, I still thought we would see the game through though.
"We kept pressing them and denying them time on the ball.
"We were very professional in our approach but just one careless bit of play in the midfield right at the end has cost us.
"That hurts us, obviously, but that is the difference sometimes between winning and losing.
"Our key players had good games today and our attacking players looked a real threat all afternoon.
"Normally when teams are losing at home, they throw caution to the wind and you have to sit on the edge of your sit a bit as you try to hold on.
"I didn't feel like that this time though, I couldn't see where they were going to score from. Jordan (Pickford) hasn't really had a save to make and we should be here talking about three points given the balance of the play.
"Unfortunately though, fine margins sometimes cost you in football and they have cost us today. In terms of the attitude of the players and the level of their performances this afternoon, however, I couldn't have asked for anything more."