As inferred in this paper

In principle, a phasing algorithm should be able to “learn” about desirable phasing configurations for a given study individual by pooling information across the reference panel and all other individuals in the study, and the phasing accuracy should increase with the sample size; in standard practice, most imputation methods gain phasing information about each study individual only from the reference panel, and phasing accuracy does not depend on the size of the study sample.

I would say, it is generally possible. It could be that the accuracy is lower than using a higher sample size. Nevertheless, I suggest to run for example the Imput2 pipeline and check the quality afterwards (e.g. info and certainty score).