Although a fusion reactor would indeed make helium, it appears that - at best - there would only be enough to serve its own cooling needs.
A paper by Bradshaw A.M. and Hamacher T., "Nuclear fusion and the helium supply problem", Fusion Engineering and Design (27 February 2013) noted these points:
Fusion power plants will require helium as cryogenic medium and as coolant.
High losses are expected: for a power plant like DEMO ≈ 2 t p.a.
The same power plant is expected to produce only ≈0.6 t p.a.
Global helium resources are finite: fusion will therefore exacerbate an already difficult situation.
The “back-stop” technology will be the extraction helium of helium from the atmosphere.