Ginseng (Panax ginseng C.A. Meyer) is the most famous medicinal herb in East Asia. Although medicinal components and their functions have been widely investigated, ginseng has been regarded as an underdeveloped crop in genetics and genomics research areas. This study was conducted to elucidate the structure and evolution of the ginseng genome by analyses of expressed sequence tags (ESTs) and bacterial artificial chromosome (BAC) sequences. The EST analysis based on the calculation of synonymous substitutions per synonymous site (Ks) in paralog and ortholog pairs revealed that two rounds of polyploidy events occurred in the common ancestor of ginseng and American ginseng (P. quinquefolius L.) and subsequent divergence of the two species. The sequence analysis of repeat-rich BAC clones characterized the major component of the ginseng genome, long terminal repeat retrotransposons (LTR-RTs). The LTR-RTs were classified into five main families, of which three (PgDel, PgTat, and PgAthila) belonged to Ty3/gypsy and the other two (PgTork and PgOryco) to Ty1/Copia. High abundance of the LTR-RTs were revealed by whole genome shotgun (WGS) read mapping and fluorescence in situ hybridization (FISH) analysis. Particularly, the most abundant PgDel family have played major roles in expanding heterochromatic regions as well as remodeling euchromatic regions. Biased intensity of the PgDel2 FISH signals on half the total chromosomes demonstrates the allopolyploid feature of ginseng genome. Insertion time estimation of each LTR-RT implied that LTR-RTs have proliferated after the recent polyploidization of ginseng. These results suggest that the ginseng genome of the present day has been expanded and evolved by two rounds of polyploidization and accumulation of LTR-RTs.