Okay, so what could be afforded? Getting consistent, comparable figures in a hurry is a bit of a nightmare, and the economy of a Scottish State is somewhat speculative, however these seem to be in the right ball park. North Sea oil is a HUGE unknown factor. If Scotland were to get all the oil and gas that's geographically inside it's territorial waters, it would get 90% of UK production. However if the oil and gas were split up by relative population, they'd get 9%. I can't see Westminster agreeing to the former, so something in between is more likely. However, let's take a "best case" (for Scotland) scenario.
Gross Domestic Product (GDP):
Netherlands : $770 billion
Switzerland : $633 billion
Norway : $500 billion
Belgium : $480 billion
Austria : $400 billion
Denmark : $315 billion
Finland : $250 billion
Portugal : $212 billion
Ireland : $211 billion
New Zealand : $170 billion
Scotland : around the $220 billion mark, assuming they get all the oil and gas in their waters.
So the comparisons being bandied about between an independent Scotland and "small" countries like Norway and Belgium are spurious. Even on a highly optimistic figure, Scotland is more in the ball park of Ireland. Furthermore, this takes into account neither the short-term economic disruption caused by a declaration of independence, nor the long-term consequences if large companies, particularly defence ones, elect to move south of the border to service the larger UK market.
Next question is what might their defence budget be? Well that's a political choice, but given the politics and society of the country, it's likely to fall within European norms of between 2.6% (UK) and Ireland (0.6%). The SNP are also committed to expensive social programs and there isn't much public perception of a threat, so I think it's unlikely to be above 2% in practice. However, best case:
Defence Budget:
Netherlands : $11 billion
Norway : $7.1 billion
Belgium : $5.4 billion
Portugal : $5.2 billion
Denmark : $4.6 billion
Switzerland : $4.4 billion
Finland : $3.7 billion
Austria : $3.5 billion
Ireland : $1.4 billion
New Zealand : $1.35 billion
Scotland : between $3.3 billion (1.5% GDP) and $5.5 billion (2.5% GDP)
So on the face of it, Scotland could have armed forces of somewhere between Austria and Belgium in size. HOWEVER, what that doesn't take into account is start up costs. Scotland is lacking a great deal of the infrastructure neccessary to run armed forces that other states have built and paid for long ago. It has no defence ministry, no defence procurement system, no general staff, no military headquarters, no staff college and no logistics system.
Note that the RUSI report figures are rather lower (£2 billion: about $3.1 billion) but they're for a hypothetical 2010 Scottish defence budget, and remember, the GDP figure I've used is top-end.