IANAL, but I think the general answer is no. When someone contribute code to an open source project, although they aren’t giving up their copyright, they do grant the recipient (and the rest of the world, for that matter) a license to use their code. In case of Linux, this is the GNU Public License. Unless GPL has a section about license revocation that I am not aware of, you won’t be able to take your code back.