The word bee, as used in spelling bee, is one of those language puzzles that has never been satisfactorily accounted for. A fairly old and widely-used word, it refers to a community social gathering at which friends and neighbors join together in a single activity (sewing, quilting, barn raising, etc.) usually to help one person or family.
The earliest known example in print is a spinning bee, in 1769. Other early occurrences are husking bee (1816), apple bee (1827), and logging bee (1836). Spelling bee is apparently an American term. It first appeared in print in 1875, but it seems certain that the word was used orally for several years before that.

Those who used the word, including most early students of language, assumed that it was the same word as referred to the insect. They thought that this particular meaning had probably been inspired by the obvious similarity between these human gatherings and the industrious, social nature of a beehive. But in recent years scholars have rejected this explanation, suggesting instead that this bee is a completely different word.
One possibility is that it comes from the Middle English word bene, which means "a prayer" or "a favor" (and is related to the more familiar word boon). In England, a dialect form of this word, been or bean, referred to "voluntary help given by neighbors toward the accomplishment of a particular task." (Webster's Third New International Dictionary).
Bee may simply be a shortened form of been, but no one is entirely certain.

References:

A Dictionary of American English. Sir William A. Craigie and James R. Hulbert, eds. University of Chicago Press, 1944.

A Dictionary of Americanisms. Mitford M. Matthews, ed. Univ. of Chicago Press, 1951.

Mencken, H.L. The American Language. New York: Alfred Knopf, 1938 (suppl. I, 1945: suppl. II, 1948).