As far as warhead mass goes, Anthony Jackson says the theoretical limit on mass for a fusion warhead is about 1 kilogram per megaton. No real-world system will come anywhere close to that, The US W87 thermonuclear warhead has a density of about 500 kilograms per megaton. Presumably a futuristic warhead would have a density between 500 and 1 kg/Mt. Calculating the explosive yield of a weapon is a little tricky.
For missiles, consider the US Trident missile. Approximately a cylinder 13.41 m in length by 1.055 m in radius, which makes it about 47 cubic meters. Mass of 58,500 kg, giving it a density of 1250 kg/m3. The mass includes eight warheads of approximately 160 kg each.
Wildly extrapolating far beyond the available data, one could naively divide the missile mass by the number of warheads, and divide the result by the mass of an individual warhead. The bottom line would be that a warhead of mass X kilograms would require a missile of mass 45 * Xkilograms, and a volume of 0.036 * X cubic meters (0.036 = 45 / 1250). Again futuristic technology would reduce this somewhat.
Nuclear weapons will destroy a ship if they detonate exceedingly close to it. But if it is further away than about a kilometer, it won't do much more than singe the paint job and blind a few sensors. And in space a kilometer is pretty close range.
Please understand: I am NOT saying that nuclear warheads are ineffective. I am saying that the amount of damage they inflict falls off very rapidly with increasing range. At least much more rapidly than with the same sized warhead detonated in an atmosphere.
But if the nuke goes off one meter from your ship, your ship will probably be vaporized. Atmosphere or no.
[1] George William Herbert says a nuke going off on Terra has most of the x-ray emission absorbed by the atmosphere, and transformed into the first fireball and the blast wave. There ain't no atmosphere in space so the nuclear explosion is light on blast and heavy on x-rays. In fact, almost 90% of the bomb energy will appear as x-rays behaving as if they are from a point source (specifically 80% soft X-rays and 10% gamma), and subject to the good old inverse square law (i.e., the intensity will fall off very quickly with range). The remaining 10% will be neutrons.
The fireball and blast wave is why nuclear warheads detonating in the atmosphere will flatten buildings for tens of kilometers, but detonations in space have a damage range under one kilometer.
For an enhanced radiation weapon (AKA "Neutron Bomb") figures are harder to come by. The best guess figure I've managed to find was up to a maximum of 80% neutrons and 20% x-rays.
If you want to get more bang for your buck, there is a possibility of making nuclear shaped charges. Instead of wasting their blast on a spherical surface, it can be directed at the target spacecraft. This will reduce the surface area of the blast, thus increasing the value for kiloJoules per square meter.
According to John Schilling, with current technology, the smallest nuclear warhead would probably be under a kiloton, and mass about twenty kilograms. A one-megaton warhead would be about a metric ton, though that could be reduced by about half with advanced technology.
Eric Rozier has an on-line calculator for nuclear weapons. Eric Henry has a spreadsheet that does nuclear blast calculations, including shaped charges, on his website. For bomb blasts on the surface of the Earth or other planet with an atmosphere, you can use the handy-dandy Nuclear Bomb Effects Computer. But if you really want to do it in 1950's Atomic Rocket Retro style, make your own do-it-yourself Nuclear Bomb Slide Rule!