Yes it certainly is a private pyrotechnics firm that provides the display but their costs are met almost entirely by the taxpayer.
I cannot find a detailed breakdown of costs for recent events but I did find an article which provided a FOI request relating to the 2008 display:
-------------------------------
A breakdown of the source of funding for the display
the £1.7m total cost of the display was met from the following sources:
London Development Agency (i.e. the taxpayer) £1.414m
Greater London Authority (i.e. the taxpayer) £186k
LG Sponsorship (no idea what that is) £100k
-------------------------------
Also, boroughs (or rather, their taxpayers) adjacent to the display area (notably Camden and Lambeth) allocate substantial sums to assist with the clear up. Numerous attempts have been made to attract commercial sponsors to bear the brunt of the costs but these seems to have been unsuccessful. This is hardly surprising as, in 2012, a commercial sponsor was found but they (who were unnamed) pulled out six weeks before NYE leaving the Mayor (i.e. the taxpayer, again) to pick up the bill.
The Mayor is very generous with other people's money for events such as this (which began under the mayorship of one K. Livingstone). Many people in the some parts of outer London (who contribute heftily to the GLA's funds) have not a hope in hell of attending the event they pay for as they have no realistic way to get home from central London after midnight