1861 census help

[Jennifer Eccles]

can anyone help with a pickle i have got myself in please,

I am tracing back through the census' John Siddons 1861 born in West Bromwich,
His parents are Alexander Siddons 1823 West Bromwich and Jane (Gould) 1826.

Now i have him in 1891 , 1881 and 1871. But nowhere to be seen in 1861, but thats ok because he was only born that year.. but.. he had two older siblings,

William Siddons 1857 West Bromwich
sarah Anne Siddons 1859 West Bromwich
both should be on the 1861 census, but cant be found.

going then to search for his father Alexander Siddons 1823,He is on the 1851 census living at home with his mum Mary Anne Siddons, and siblings William 1827, Mary 1829, and Sarah A 1831. (Still all West Bromwich)

Mary Anne is missing from the 1861 onwards so presumes she is dead before this, as is Alexanders father before 1851.

Now, Alexander 1823, was married before 1861 as he had william A in 1857 and sarah in 1859, but neither him nor the children nor his wife can be found in 1861 anywhere as far as i can see in the Uk.

I have found William, Alexanders brother in the 1861, still in West Bromwich now married with children, ( he married an Anne E, b Denbighshire) and John (1861) is in West Bromwich in 1871 onwards.. so i cant see why his father Alexander and family would have moved somewhere then moved back?