Rogers, ranked 43rd in the world, took the first set, but Barty bounced back to level the match at a set apiece. Barty led the third set 5-2 and was serving for the match before Rogers mounted a furious comeback to force a decisive tiebreak.
Rogers won the tiebreak to stun the crowd at Arthur Ashe Stadium at the USTA Billie Jean King National Tennis Center.
Rogers, 28, moves on to the round of 16 at the US Open for the second straight year. She reached the quarterfinals a year ago.
She will face Britain’s Emma Raducanu in the round of 16 on Monday. The 18-year-old Raducanu beat Sara Sorribes Tormo of Spain 6-0, 6-1 earlier Saturday.