What I don't like about Apple devices is that whenever I have trouble with them, there's very few things to do. With PCs, there's always at least half a dozen options, ranging from rebooting to tinkering in the registry, but with every Apple problem I've had it's always power off and back on, do a restore, and go to the genius bar, who inevitably tell me they can't do hardware support because isn't a new device.
That's my second annoyance - hardware support and spare parts simply cannot be obtained through official channels for non-new devices, not even for money. In contrast, Dell is quite happy to sell me spare parts and tech support for anything I've ever bought from them, even if it's out of warranty. They don't always have the best prices, but they're at least willing to provide it. At Apple that isn't even an option - the best they can do is give you a discount on a new device or a replacement of the same kind of device.
This focus on novelty also extends, most irritatingly, to software and operating systems. If you restore your ipod, it automatically installs the latest software, and there's no possibility of rolling it back. If one of your apps isn't compatible with the new iOS or it's otherwise worse than the previous version, you're stuck. In comparison, Windows lets you uninstall any updates and service packs without even having to do a system restore (although that's totally an option). I could even take my old Windows 98 CD and install Windows 98 on my current computer. Microsoft wouldn't support it any more, but it's not like they have technological measures in place to stop me.
Apple's general philosophy seems to be that the products are intended to just work without the end user having to worry about fixing them. But I've had my fair share of problems, and not being able to get at the guts like I can with my PCs is irritating.