使用Podman创建Ubuntu容器后无法通过SSH密钥登录问题求助

我最近尝试用Podman创建一个Ubuntu容器，容器本身能正常启动，但当我想用SSH密钥登录进去的时候却一直失败，实在搞不懂哪里出问题了，想请大家帮忙看看。

下面是我用到的Dockerfile内容：

FROM ubuntu:latest
RUN apt update && apt install  openssh-server sudo -y
# Create a user “sshuser” and group “sshgroup”
RUN groupadd sshgroup && useradd -ms /bin/bash -g sshgroup sshuser
# Create sshuser directory in home
RUN mkdir -p /home/sshuser/.ssh
# Copy the ssh public key in the authorized_keys file. The idkey.pub below is a public key file you get from ssh-keygen. They are under ~/.ssh directory by default.
COPY id_rsa.pub /home/sshuser/.ssh/authorized_keys
# change ownership of the key file. 
RUN chown sshuser:sshgroup /home/sshuser/.ssh/authorized_keys && chmod 600 /home/sshuser/.ssh/authorized_keys
# Start SSH service
RUN service ssh start
# Expose docker port 22
EXPOSE 22
CMD ["/usr/sbin/sshd","-D"]

我尝试用SSH密钥登录的时候就出现了问题，目前还没找到原因，有没有大佬能指点一下？

备注：内容来源于stack exchange，提问作者Madhuri