Mouse is one of the oldest words in the language and barely changed in thousands of years. Old English had mus, and the same form turns up as Sanskrit mus, Old Persian mush, Latin mus and Greek mys — the Indo-European name for the creature, retained almost intact across families that separated before writing existed.
The plural mice is the point of interest for spelling. It is not irregular so much as a fossil of an older grammar: Old English mys shows i-mutation, a sound change in which a vowel in the ending pulled the vowel in the root forward, and then the ending fell away leaving only the altered vowel behind. Goose and geese, foot and feet, man and men are the same process. A curiosity: the Greek and Old English words also meant "muscle," from the resemblance of a flexing bicep to a mouse under the skin — which is where muscle itself comes from.
mouse (n.) Middle English mous , from Old English mus "small rodent," also "muscle of the arm" (compare muscle (n.)); from Proto-Germanic *mus (source also of Old Norse, Old Frisian, Middle Dutch, Danish, Swedish mus , Dutch muis , German Maus "mouse"). This is from PIE *mus- , the old Indo-European name of the creature, retained in several language families (source also of Sanskrit mus "mouse, rat," Old Persian mush "mouse," Old Church Slavonic mysu , Latin mus , Lithuanian muse "mouse," Greek mys "mouse, muscle"). Plural form mice (Old English mys ) shows effects of i-mutation . As a type of something timid or weak, from late 14c. Contrasted with man (n.) from 1620s ( nor man nor mouse ). The meaning "black eye" (or other discolored lump on the body) is attested by 1842. The computer sense of "small device moved by the hand over a flat surface to maneuver a cursor or arrow on a display screen" is from 1965, though the word has been applied to other devices resembling a mouse in shape since 1750, mainly in nautical use. Parturient montes, nascetur ridiculus mus [Horace] mouse (v.) "to hunt or catch mice," mid-13c., mousen , from mouse (n.). Related: Moused ; mousing . also from mid-13c.