When Virginia seceded in 1861, the Confederate government found Belle Isle an ideal place for a prisoner of war camp. As many as 30,000 Union prisoners of war were held there between 1862 and 1865, peaking at 10,000 in 1863 with tents for only a third of the population. Conditions were horrible. Many attribute unusual activity on the island to the ghosts of these Union prisoners of war.