I always used rails. DD1 is 4.5 and my DH is like "You can probably take the rails off", but her bed is a queen sized and pretty high up (she needs to use a stool to get into it.
Having said that, rails are often not installed correctly, and come loose, so I am not sure how much protection they offer anyway.