Bug 4. (eyes) To grow large, especially in bulging outward.
What meaning of in is used here?
On the one hand, I want to say that this seems like an error in your dictionary. It isn't possible to explain the function of a word that probably shouldn't be there.
On the other hand, I think it is worth saying that dictionary definitions do not always follow standard English grammar. The language used in dictionary definitions is sometimes called defining language. The book Defining Language: A Local Grammar of Definition Sentences by Geoff Barnbrook says that definition language meets the criteria of being a sublanguage model. So, it shouldn't really be critiqued by the normal standards of English grammar.
Regardless of whether it is an error or not, I certainly understood it. Did you? In a dictionary definition, that is all that really matters.