I joined the Apple ship a couple of years ago with an iPad2 and soon after got a 4S to replace my HTC Desire. Last year I jumped Apple for Nexus, 4 and 7. Now I'm back to iPad Mini and Nexus 7, with an iPhone 5 arriving (work phone) on Monday.
If you've used Android and taken the time to use the phone to anything like it's potential, then it can feel like a step backwards to go to Apple. There are (in my experience) two main reasons for this:
1) The amount of customisation on an Android device is far in a way superior to that on an Apple. There are widgets for a start, and a central home page to 5 app screens. This is compared to a left hand home screen and no visual customisation other than the order of the apps on the screen (and they 'ping' to fill all the gaps) and the wallpaper.
2) Android phones can run apps in the background. This to me, from a performance perspective is the main downside to Apple
Having said that, Apple stuff does just 'work' and is in general more user friendly than Android, although the gap is closing. The other major advantage Apple has over Android is that because Apple works on a sandboxed design for its apps, generally speaking it will always more secure than Android. Plus Android is become the mobile OS of choice for malicious malware which has apparently risen by 614% over the last year (albeit largely from apps downloaded from unofficial app store copy sites).
Apple, in selecting the apps and validating them as part of the app store process means that the number of 'dodgy' apps is far reduced compared to Android, but again something the Android Play store is also working on and has improved significantly. Apple do though filter what can be sold as an app and do not permit anything that can be classified as a hacking tool, so you can't get decent wireless analysers or netscan tools (as of last time I looked, if this has changed will be stood corrected). These are very useful tools and it frustrated me that I was told what I can and can't do with my device. As an IT Security consultant, those can be useful tools to have to hand.
By choice, I'd still stay Android. Work uses Apple for our phones/tablets so it's come back into the equation for that reason alone. My current Apple experience is now better than my previous, and there are good reasons for sticking with it. The Google/Apple integration is better than it's been in the past. I think mostly the smartphone OS debate is becoming a more level playing field. Android is now a universal OS whereas a couple of years ago, it was kit for the geeks among us to figure out. Apple is becoming less restrictive (in that it's had to improve support for non-apple apps/platforms like Gmail, Exhange) and they will always be nice pieces of technology, for build quality if nothing else.