How are return checks processed on the system?
That depends on how you want to handle them. They could remain as pending bank transactions if you are going to resubmit them to the bank. You could delete them if there is no hope of them being honored. From an accounting viewpoint, a check that is not good is no payment at all.
You should not do this if your Bank Statement shows the cheque being deposited and its subsequent dishonouring as your manger accounts would not exactly reflect your bank account activities.
To do a returned cheque on the system, (assuming it was from a customer) do a Spend Money with the account being Accounts Receivable + Customer + Invoice, then any Customer Statement produced will have the full customer activity detailed. This documentation of the events is essential.