The company was founded in 1963 by Arthur Benjamin Sugarman (1925-1987), who was born in Brighton. He emigrated to the United States in 1946, via Canada, and changed his nationality to American. He married the daughter of a Californian clothes producer and later returned to Brighton, where he bought a shirt factory. He lived in Ditchling Road with his family opposite Hollingbury Park near the golf course. Sugarman had realised that early 1960s London-based modern jazz fans were eagerly buying the Oxford-cloth American button-down shirt brands such as Brooks Brothers, Arrow, and Hathaway, that were worn by visiting American jazz artists like Miles Davis, Dizzy Gillespie, and Oscar Peterson. At the time, these were only available from official importers who had in effect, cornered this market.
Sugarman decided to produce a version of these shirts, along with a collection of the colourful, resort-wear vacation clothes that were growing in popularity, in both the U.S. and Mediterranean Europe. Mods responded, especially as Sugarman was using higher-quality materials and stitching detail than the imported shirts. The Ben Sherman Originals label was created, and by 1965, the company had opened a small office on the upper floors of an office-block in a London backstreet. This acted as the showroom for their shirt and beachwear collections. The first Ben Sherman store was opened in Brighton in 1967.
Sugarman sold his business in 1975 and moved to Australia. The company passed through a number of hands in the following years. In 1993, British investor 3i backed a management buyout of Ben Sherman Limited from Northern Ireland based Dunkeld fashion group, then in receivership, for 4 million pounds which resulted in the creation of the Sherman Cooper Group. In 2000, 3i financed a second management-buyout that created Ben Sherman PLC. In mid-2004, Oxford Industries Inc. of Atlanta purchased the Ben Sherman brand for 80 million British pounds (then $146 million) from 3i and Irish venture capital company Enterprise Equity.
The Ben Sherman womenswear and childrenswear collections that existed in the 2000s were discontinued by Oxford Industries in 2010. On 11 October 2010, the Ben Sherman opened a store at №39 Savile Row. In 2011, the company launched its Plectrumcollection of better menswear. On 31 October 2011, Ben Sherman launched its new eCommerce website.