Off-the-shelf software is usually faster and cheaper to implement, and suits processes that are fairly standard across businesses. Custom software becomes worth considering once a business is working around the limitations of generic tools with manual workarounds or spreadsheets.
A useful test is to map the actual workflow and count how many steps involve manual data re-entry, workarounds, or tools that do not talk to each other. The more friction that mapping reveals, the stronger the case for a tailored solution.
Many businesses land on a hybrid approach — off-the-shelf tools for standard functions like accounting, and custom software for the specific processes that make the business distinctive.