The landmark 12-country Trans-Pacific Partnership trade deal could bring more business to Vietnamese manufacturers, but there is a potential downside: higher labor costs and strained production at the country’s factories.

Over the past decade, manufacturing labor costs in Vietnam have more than tripled to $1.96 an hour, including benefits, although costs are still much less than China’s $3.27 an hour and the U.S.’s $37.96 an hour, according to the Economist Intelligence Unit.

Stanley Szeto, the chief executive of Lever Style, a Hong Kong-based firm that manufactures shirts and pants for brands from Hugo Boss to J. Crew, says he’s “not very excited” about the trade pact because any surge of investment could make it more expensive to manufacture in Vietnam.

“Costs will go up and it’ll be harder for everybody to get capacity” at factories, says Mr. Szeto, a former investment banker who took over the family business from his father in 2000. “Labor (procurement) is going to be more competitive.”

In the past five years, Lever Style has moved a quarter of its production from China to Vietnam, paring its China workforce nearly 50% to 3,000 employees in the process. The manufacturer says it hasn’t been able to move to Vietnam faster because factories in the country still lack the know-how to manufacture a sophisticated jacket or shirt.

Although the TPP is expected to eliminate tariffs between members on items such as clothing, contract manufacturers in Vietnam are likely to see little of those savings if the pact goes through, says Mr. Szeto.

That is because generally, global brands—not manufacturers—pay the cost of import duties under an arrangement where responsibility for the goods passes to the buyer once the products are shipped. So the elimination of tariffs reduces costs for brands rather than manufacturers.

Manufacturers can negotiate with global brands a higher price for production once the trade deal goes through to share in the cost savings, according to Roger Lee, chief executive of TAL Group, which makes 1 of every 6 dress shirts sold in the U.S. for brands from Banana Republic to Brooks Brothers.

But global brands would only consider this if they have a large amount of production concentrated with one manufacturer, according to Mr. Lee.

Still, manufacturers in Vietnam would benefit from the TPP because global brands will likely choose to source more from the country, boosting volume for factory operations there, says Adam Sitkoff, executive director of the American Chamber of Commerce in Hanoi, a membership group for U.S. companies.

“The idea that saving 10 cents because of lower tariffs means the 10 cents just goes into pockets of the global brands is not really how it works,” says Mr. Sitkoff, noting that manufacturers benefit from increased production in the country.

Avery Dennison, a Glendale, Calif.-based producer of labels and packaging materials that opened a 300,000 square-foot facility in southern Vietnam this year, says it doesn’t see the trade pact as a way to cut its costs in the country.

“It has always been about the growth opportunity it creates and ensuring that we are in a competitive position to capitalize on the opportunity,” says Frank Smigelski, a vice president in Avery Dennison’s retail branding business.

By Kathy Chu - The Wall Street Journal - October 18,2015