I can think of a couple of possible reason.
Do you have enough of the right fruit at the markets? Breweries need a lot of fruit. The brewers get 100 fruit at the time, if it's available. If there's less they must walk often (or far away) to get some and the productivity goes down.
Do you get rid of the produced alcohol? Ale is a bit special; it is not delivered to markets and barns, the people drink it at the tavern. If there is enough alcohol in the store, nothing new will be produced. The same if you sell the ale. In this case the traders get the ale at the breweries. If the traders are too busy, they might not be able to get everything and the productivity gets down.. This is the problem if you trade a lot of ale, especially in combination with other things. It seems like everything else is collected first. I have seen traders pass breweries full of ale to get 8(!) mushrooms.
These are the one I know at the moment. But breweries are strange. If you look at my picture. I cannot explain the difference .