You may find the fan difficult to remove. One tip is to apply the prying force in line with the shaft (vertically) as much as possible.

You may find it easier to break the fan free by rotating the screwdriver at first to pry upward.

As the fan starts to slide on the shaft, you may also want to put the screwdriver tip further from the hub. Then lift with the shank as you start to remove the fan.