There are two main issues with a site's design. One is the user interface. Does the design allow visitors to find what they're looking for with minimal friction? This sounds a lot easier than it really is. I've regularly had people tell me that their site "works" because they have subscribers or sales or whatever the metric for success happens to be. However, with a bad design, only those who are obsessive about the content will put in the effort to dig through a bad user interface. At the same time, people who might be interested in what you have to offer will never know because they won't be able to find it.
The other issue is branding. Are the visuals congruent with the message the site is trying to convey? There are a zillion considerations within this question. Is it a make or break issue? Maybe. Does it make a difference overall? Yes.