Reference decision is a crucial downside, one that’s important to know and efficiently deal with contexts of various varieties. This context contains each earlier turns and context that pertains to non-conversational entities, similar to entities on the person’s display screen or these working within the background. Whereas LLMs have been proven to be extraordinarily highly effective for a wide range of duties, their use in reference decision, significantly for non-conversational entities, stays underutilized. This paper demonstrates how LLMs can be utilized to create an efficient system to resolve references of assorted sorts, by displaying how reference decision will be transformed right into a language modeling downside, regardless of involving types of entities like these on display screen that aren’t historically conducive to being diminished to a text-only modality. We exhibit massive enhancements over an present system with related performance throughout several types of references, with our smallest mannequin acquiring absolute positive factors of over 5% for on-screen references. We additionally benchmark in opposition to GPT-3.5 and GPT-4, with our smallest mannequin reaching efficiency similar to that of GPT-4, and our bigger fashions considerably outperforming it.