Shatteredragon's got some great ideas, Drammatex. I agree with him expecially on one main front: decide what use you're going to have from this castle first, and then build it after planning it out. (heh. It's the one problem of having such a wonderfully modular design!)

If you're doing an infiltration adventure, you might need a single tower, a few walls, a courtyard, and the buildings.

If you're doing a full-scale parade through the main gatehouse, you'll likely need quite a bit more.

You can always start with a small adventure, and build out from there!