Hi!
I'm traveling down this path a little before you, but both of us are about 25 years late to the party. So I'll chime in with my experiences so far and try to be helpful.
First off, you didn't say if you were doing TOS (kirk) or TNG/DS9 (picard). This will influence which ship scale you're gonna chase like a space crack addict. I'm doing TNG era. I have an amusing blog post about starting ST gaming here if interested:
link
Neither era is wrong, but TNG is more right. If you're going the TOS route than ADB can be your one stop shopping for miniatures. Just pick the scale you like. They also have the SFB and Fed commander rule sets. But be warned: rules for the TOS era seemed to be have been produced after the show ended but before the movies and are set in a world that to me is more ‘star trek adjacent.' Ships will be firing rockets and shuttles at each other…
If going the more proper TNG route, there is NO dedicated miniatures range so you have to cobble together your own from a variety of sources / shapeways store. I looked at a LOT of scales for ships (and the sizes are hugely varied) but ultimately settled on 1/7000; mainly because I was inspired by this gentleman's gallery:
link
and ended up just copying what he did (imitation being the sincerest form of flattery). and to plug myself again; here is another humorous blog post I wrote about scales in Star trek:
link
Which means buying a lot of the models off of Shapeways.
As for rules, besides some of the oldies but goodies mentioned in replies above, there really isn't a rule set specifically for ST. So the best bet is to find a generic rule set that lets you design your own ships and create them yourself. I like a small fleet game (10 ish ships per player) and I would recommend:
Full thrust Cross Dimensions: can be found free here
link
There are several different version of FT; this is the one I like. Design the ships as you want and go shoot things. I thought this would be my go to rule set but then I stumbled upon Starmada Unity (on Wargame vault).
There are several early editions to Starmada with all sorts of names, Starmada Unity is the latest. It has a little more crunch to it than FT and therefore provides a few more options / dials you can set and variety but is not overwhelming. I've played a few games of Klingon Armada Unity (which is the Starmada in the TOS era trying to play like SFB or fed commander) and can say the core rules are lots of fun.
Some caveats though: the standard momentum / engine requirement rules are stupid and hard; but there are many better options for it included in the optional rules that make it elegant.
Ship design in Starmada Unity is super easy ONCE YOU FIGURE OUT HOW TO USE THE DAMN SPREADSHEET. You can get the drydock ship builder off the website. Without the spreadsheet ship design is impossible.
Last comments;
The Fleet captains board game is indeed, excellent. Best game of ST hands down.
Parizal's GOBS book is just fun to read.
Hope any of that was helpful. I just actually finished painting up some ST ships myself.
-Stew