Promoting Cross-Modal Representations to Improve Multimodal Foundation Models for Physiological Signals

Many healthcare purposes are inherently multimodal, involving a number of physiological alerts. As sensors for these alerts turn out to be extra widespread, enhancing machine studying strategies for multimodal healthcare information is essential. Pretraining basis fashions is a promising avenue for achievement. Nevertheless, strategies for growing basis fashions in healthcare are nonetheless in early exploration and it’s unclear which pretraining methods are handiest given the range of physiological alerts. That is partly resulting from challenges in multimodal well being information: acquiring information throughout many sufferers is tough and expensive, there may be loads of inter-subject variability, and modalities are sometimes heterogeneously informative throughout downstream duties. Right here, we discover these challenges within the PhysioNet 2018 dataset. We use a masked autoencoding goal to pretrain a multimodal mannequin. We present that the mannequin learns representations that may be linearly probed for a various set of downstream duties. We hypothesize that cross-modal reconstruction targets are necessary for profitable multimodal coaching, as they encourage the mannequin to combine info throughout modalities. We exhibit that modality dropout within the enter area improves efficiency throughout downstream duties. We additionally discover that late-fusion fashions pretrained with contrastive studying targets are much less efficient throughout a number of duties. Lastly, we analyze the mannequin’s representations, displaying that spotlight weights turn out to be extra cross-modal and temporally aligned with our pretraining technique. The realized embeddings additionally turn out to be extra distributed by way of the modalities encoded by every unit. General, our work demonstrates the utility of multimodal basis fashions with well being information, even throughout numerous physiological information sources. We additional argue that specific strategies for inducing cross-modality could improve multimodal pretraining methods.