Dr Brian Dunn was speaking a year after the pandemic started.
In Northern Ireland, 18 deaths were related to swine flu. Two other people from Northern Ireland died abroad from swine-flu related illness.
The cost to the Department of Health is £44m. "I would not like to be accounting for that," Dr Dunn said.
"At a time when money is scarce and is likely to get scarcer; at a time when waiting lists are going up; at a time when we are talking about cuts in front-line services, I think it is very hard to justify that."
In February, it emerged that half a million swine flu vaccines remained unused in Northern Ireland.
But at that time, another member of the British Medical Association argued it was better to have too many than too few vaccines.
Dr Tom Black said swine flu no longer posed a threat - therefore, lives have been saved.
"I think it is better to have vaccine left over than to be left short," he said.
"If we had needed to vaccinate the whole population then we would have needed a lot more vaccine.
"This was the fourth pandemic influenza outbreak in the last century. The previous three killed millions of people. I think we did very well with this one."
The health minister has insisted that all decisions related to the virus were taken after scientific advice.
Michael McGimpsey agreed with fellow executive ministers in December that he would need £61m to combat the threat of the virus.
The death rate from those suffering from the virus across the UK was low, at 0.026%. In addition, hundreds of thousands of people are thought to have been infected but not displayed any symptoms.