Halep served for the match at 5-3 in the final set and had a match point in the next game, but the French Open champion couldn't capitalize on her chances.
In that game, Hsieh got off to a good start, winning a tight opening set in a tiebreak, but it was Halep who prevailed, winning 6-7 6-1 6-2.
"I knew she [was] going to fight really hard", Hsieh said.
Halep, the French Open champion, looked like she would escape after forcing match point at 5-4 against serve in the third, but Hsieh rallied before breaking again en route to serving out the victory.
Gulbis, ranked 138 but a former world number 10 who famously defeated Roger Federer on his way to the French Open semi-finals in 2014, will be playing in the last 16 at Wimbledon for the first time. She stayed there for every point.
A former world No. 1 in doubles and the 2013 Wimbledon doubles champ, the 32-year-old is the first Taiwanese player, male or female, to break into the top 25 in the singles rankings and the first to hold the top ranking in doubles.
Simona Halep of Romania loses a point to Su-Wei Hsieh of Taiwan during their womens singles match on the sixth day at the Wimbledon Tennis Championships in London, Saturday.
I just was too negative to myself, talking too much. "I would never throw the match".
Bencic, now at 56 in the world, next faces two-time Grand Slam champion Angelique Kerber who swept past Naomi Osaka of Japan.
Hsieh continued posing puzzles though and broke in the third game of the second set as she absorbed a fierce baseline barrage before punishing a clumsy Halep volley by spearing a backhand down the line to roars from the crowd.
Making 10th main draw appearance at Wimbledon, Su-Wei said her relaxed approach to the season has been crucial to her progress to the fourth round for the first time.
East European 21-year-olds Jelena Ostapenko and Daria Kasatkina carried the flag for the Wimbledon seeds with straight-set victories on Court Three that took them through to the fourth round on Saturday.
For the first time at Wimbledon, none of the top five women's seeds reached the round of 16. I think because I was exhausted, because I'm exhausted, I couldn't stay focused on every ball.
"I feel I'm really lucky to be a tennis player, stay here until the second week".
The Serb, seeded 12th, will meet big-hitting Russian youngster Karen Khachanov for a place in the quarter-final.
Otherwise, Rafael Nadal, Novak Djokovic and Juan Martin del Potro all won on a day the English spectators were preoccupied with two things: soccer and the sun. She wouldn't win another game.
The Belgian, ranked number 47 in the world, knocked out defending champion Garbine Muguruza in round two.
She said: "One is enough".
She will now play Dominika Cibulkova after the Slovak, who defeated Britain's Johanna Konta in the second round, beat 15th seed Elise Mertens of Belgium 6-2 6-2.