Say if I had a custom line on a invoice which I need to multiply the quantity and the unit price to get the amount , can this be done?
Calculation within fields does not work in custom fields. It works in other number fields. See this Guide: Perform calculations in number fields | Manager.
But why would you want to multiply quantity and price except in a line item? What exactly are you referring to as a “custom line?”