I was actually going to post a few ideas for small villages, might as well do it here instead making a new thread. I'm not sure if this is what you're looking for, but perhaps it can help.
After completing vanilla achievements and playing with mods I found out I prefer really tiny settlements (up to 50-60 people) that grow rather slowly. I came up with a few challenges/ways to play that could make those villages more unique. Some of them could also work in larger towns. They do not necessarily add any difficulty, but can help with making the next playthrough differ more from the previous one. I have not played all of them so some might not actually work as expected.
Jewel of the forest
You can only build enough houses for the starting families + a single boarding house. Everyone should be equipped with gemstone tools, however you cannot trade for polished gemstones or the tools directly. Rough gemstones are OK to trade for. Ideally they should be mined locally, but their drop rate is so low that a fully staffed precious mine doesn't even guarantee a single gemstone per year, so trading is unfortuately needed. (Minimum mods: CC and Blackmith Tools from Black Liquid Software, for the materials and tools.)
They keep coming
Immigration must be your only way to increase population. To do this, you will need houses that can only hold up to 2 people (so no new citizens can be born), but an existing family can also move into them, so housing starting families should not be a problem. (Minimum mods: DS Jetty & Bridge or other mods that include houses for 2 occupants.)
Bloodlines
Pick a family and follow their descendants from the start of the game up to the 5th generation. (Minimum mods: None. A 1:1 ageing mod can make keeping track of people easier, however it does take a long time to reach further generations, so vanilla might be preferred depending on playstyle.)
Less is more
As soon as your village is stable enough to survive, pick a single production chain / item category / resource and produce only that, trading for everything else necessary to sustain the village.