Your chimney works on the very simple principle that warm air rises and cold air sinks. This is great when you have a fire burning, enabling the smoke and fumes to escape into the atmosphere.
It's not so great when you don't have a fire burning however. The same prinicipal is still working. Your warm room air is escaping up the chimney stack, and cold outside air is coming down the chimney, causing draughts.
The solution is to block your chimney when you don't have a fire burning. You will stop draughts, stop losing warm room air up your chimney, and save energy.