Where a serious search for the rightsholder is unsuccessful, their work may not be used from a strictly legal point of view. Fortunately, the copyright exceptions sometimes apply to certain uses in education.
If no exception applies, works for which the rightsholders can't be found (orphan works) are often used in practice. In that case, a disclaimer is added to the product or website that includes the orphan work. It states that the institution has made every effort to locate the rightsholders of the works used, but has not been successful (in all cases). Then the address the rightful rightsholders can still contact is stated. The rule of thumb is that if a rightsholder comes forward, a licence fee will still be paid.
However, you can't always use this as standard procedure: for each work to be used, you have to assess the (financial) risks, and you also have to bear in mind that you run the risk of not reaching an agreement with the rightsholder afterwards. When searching for the copyright owner, collective management organisations can be consulted.