Hi,
Well that does sound like he might possibly be homeless or lost, although he might just prefer your garden to his own! You could start by putting a few notes through doors in your local area (with a photo of the cat on if you can) to ask if the cat is theirs, to give you a call?
Does he look a bit skinny? A bit dirty like he is living rough? Try at first putting a few cat biscuits out and then retreat back indoors, and see if he comes back to eat them when you are out of sight - if he does, then see how quickly he is eating. If he gobbles them down like there's no tomorrow that could indicate he is homeless or lost.
The fact that he is spending lots of time in your garden, means if he is lost, you have a really good basis to start gaining his trust (with food) and getting him to the point of being able to eventually catch him, to (a) get a paper collar on him to find out if he is going home and if not (b) get him checked for a microchip. (a rescue organisation can help with trapping him if you can't catch him). For now I would start with testing out how he is with small amounts of food left for him. Also I would put some water out too.
Also, check lost & found registers & facebook pages to see if anyone has posted him as lost. Also take a photo and send it to your local vets & rescues so that they can check their lost cats registers.
There's more about what to do if you found a cat here:
https://www.catchat.org/index.php/cat-m ... ound#found
good luck, let us know how it goes...?