Couple things. First off, objects are not reference counted; reference counting schemes have the circular reference problem, whereby two objects refer to each other but are otherwise inaccessible, and thereby leak. .NET uses a mark-and-sweep approach which does not use ref counts.
Second, though the suggestion to use a weak reference is not terrible, it's also not a slam dunk. You are building a cache for performance reasons. (I assume that your careful, empirical, realistic research into the performance characteristics of your application has convincingly demonstrated that a caching strategy is necessary in order to achieve acceptable performance; if that's not the case, then you are making these decisions prematurely.) Every cache has to have a POLICY about when it releases its resources, otherwise it's a memory leak.
How do you know that the GC policy and your policy are equivalent policies? The GC was not designed with your specific performance needs in mind. That is, it was designed to release resources that really are garbage, not to achieve any particular performance goal you've got in mind. By delegating the decision to the GC, you give up your ability to tune your cache policy to your performance needs.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…