The narrator met the man and his wives, so they were coming the other way. The only person going to St Ives was the one telling the story. Do the sum anyway and you get 2,800 kits, cats, sacks and wives.