Do you have more then one Inventory Location ?
If no, then you can turn off the Inventory Location field under Settings.
If yes, then you need to manage with Goods Receipts unfortunately, however under the following topic lubos has commented about making changes so that the Inventory Location selection can occur on Invoices.
"I’ll probably add that option for consistency reasons. Normally when it comes to invoices, they are usually credit sales or credit purchases where payment or delivery of goods happen at different time.
But I agree, there are use cases where goods are dispatched at the same time as invoice is issued. So for these use cases it makes sense to be able to specify inventory location on invoices."