Jackson - Use custom deserializer only for specific JSON

We have a class containing multiple Sets of Longs. We want them serialized as arrays, and most clients do so.

However, we have a PHP client that creates sets such that they serialize in an odd way. A set with the number 4 comes in like this:

"setOfNumbers": {
  "4": 0
},

Naturally, Jackson complains about this being an object and not an array. What I would like is to have a custom deserializer that is only invoked if Jackson detects an object where a Set<Long> should be (and ideally only if they are contained in specific classes.)

I've tried this:

    this.addDeserializer(Set.class, new StdDelegatingDeserializer<>(new StdConverter<Map<String, Long>, Set<Long>>() {
        @Override
        public Set<Long> convert(Map<String, Long> set) {
            return parseLongs(set);
        }
    }));

The problem with this is that now it expects an object instead of an array for all Set fields. The class being deserialized is generated, so I can't add any annotations or make other changes.