I think that you are exactly right that we should be aiming at a detail of tasks that are around 1 day in duration when beginning development.
Our estimates are what often are so wrong (as you pointed out). I think that one of the biggest reasons is that we’re giving estimates like “2 weeks”. “2 weeks” isn’t really an estimate, it’s a guess, a prediction.
An estimate is something like “1-4 weeks” or “3-5 days”. It really should be a range. One big downside of single-point guesses is that the guess is likely to be treated like a promise (particularly by your customer).
There are few tools (e.g. LiquidPlanner) that handle estimating in ranges well so this can be hard. But if you always give all of your estimates in ranges (e.g. 4-6 person weeks, 1-3 days, 6-9 staff months, etc) it is quite clear that it is an estimate, not a prediction or a promise. Giving a range opens the discussion of what could happen (risks) that would push the effort towards the worst case end of the estimate or bring it in towards the best case end.
Another nice thing about ranged estimates is that when you are asked to estimate something with poorly defined requirements you can give wide ranges to communicate the uncertainty. Well defined projects with believable requirements get estimates like 5-7 weeks whereas poorly defined, nebulous requirements get estimates like 4-20 weeks.
Your customer will appreciate this because folks in other functions (e.g. marketing, operations, manufacturing,...) need some idea of when they are likely to take delivery of the software. A person or team that can accurately (not precisely) give estimates for these things gives their client a real competitive advantage.

