Since early 19th century ("Regency") dance is one of my particular specialties, I get many questions that boil down to either "what did Jane Austen dance?" or "did Jane Austen dance _____?" So let's see what I can do for a general answer.
I can divide things loosely into three categories: what we know she danced, what she might have danced, and what she didn't dance.
We can be quite certain that Jane Austen danced three different sorts of dances:
1. Country dances. Long lines of couples performing figures and progressing up and down the set. These were by far the most popular and most common dance of Austen's lifetime and would have been the overwhelming majority of the dances at any ball or evening of dancing at home. They wouldn't have looked quite like they do in today's modern English Country Dance community, however. Lively skipping steps would have been used (similar to modern Royal Scottish Country Dance Society dancing), rather than a flowing walk; when a dancer was too tired to do steps, she would have been considered no longer dancing at all, as with Fanny in Chapter 28 of Mansfield Park:
Sir Thomas, having seen her walk rather than dance down the shortening set, breathless, and with her hand at her side, gave his orders for her sitting down entirely.
Rather than everyone starting at once, dances would have called and led off by a single couple at the top; as that couple progressed down the set other couples would begin to dance, then lead off in turn as they reached the top, until all the dancers were moving. Jane Austen occasionally got to lead a dance, as she mentioned in a letter of November 20, 1800, to her sister Cassandra:
My partners were the two St. Johns, Hooper, Holder, and very prodigious Mr. Mathew, with whom I called the last, and whom I liked the best of my little stock.
This could lead to very long dances indeed (half an hour to an hour) if there were many couples in a set (I discussed this previously here) and to plenty of time to chat quietly with ones partner at the top or bottom of the set, as Austen uses to good effect with Elizabeth and Darcy in Pride and Prejudice. And since dances were generally done in pairs (as discussed here), one could spend quite some time with a single partner at a ball. In that same letter Austen mentions that she danced only nine dances (out of twelve) at that ball; that would have been two with each of the St. Johns, Hooper, and Holder, and that final one with Mr. Mathew.
Country dances of this era were invariably triple minor (though that's a modern term), meaning subgroups of three couples within a longer set were needed to perform the figures, and proper, meaning that each repetition of the dance began and ended with all the men on one side and the women on the other. Nor did the specific combinations of figures ("dances") have names; this is why, in all of her letters and novels, Austen never gives the name of any country dance. A fairly limited repertoire of figures was used, making dances very similar, and they could be danced to any tune of suitable length, rather than the tight pairings of choreography and tune that are considered standard nowadays. The couple calling the dance would both choose the tune and decide what figures to dance to it.
All of this standardization would begin to break down very soon after Austen's death, but during her lifetime country dances had surprisingly little variation.
2. Cotillions. Austen mentions having danced these in her youth in one of her letters to her niece Fanny (quoted below). A cotillion was a dance for a square of four couples and would have consisted of a specific set of figures which were repeated as a sort of chorus between a series of "changes," which were simple figures (circle, grand right and left, etc.) The changes -- usually ten in number, though I've seen sources with more or fewer -- were fairly standardized, so once one had learned them, it was easy to simply pick up a new cotillion and use its special figures as the chorus to the known "verses" of the changes. As with country dances, steps were used, rather than walking; an easy cotillion figure of the 1770s is:
The Ladies Contretems Forward and turn to face their Partners, then all Eight Allemande. All half a Course, with the Rigaudon at every Place. The Gentlemen Contretems Forward and turn to face their Ladies, then all Eight Allemande. All half a Course with the Rigaudon at every Place. (Giovanni Gallini, New Collection of Forty-Four Cotillions, London, c1770).
The Contretems and Rigaudon are specific dance steps. The above is a very simple cotillion; they could be much longer and more complex.
3. The Boulanger, also known as the Boulangere, Boulangeries, etc. This was a "finishing dance" for the end of an evening, and is actually not all that exciting to do. Couples would stand in a circle, and one dancer at a time would go around the circle turning each dancer of the opposite sex in alternation with her own partner. This would alternate with a figure where all the couples circle first one way, then the other. About all I can say for this as a dance is that it's certainly an easy one to do very late at night when exhausted and with minimal energy or concentration left. But we know for certain that she danced it, as it is mentioned in her letter of September 5, 1796, to her sister Cassandra:
We dined at Goodnestone, and in the evening danced two country-dances and the Boulangeries.
Austen also used it in Pride and Prejudice, in Mrs. Bennet's description of Mr. Bingley's activities at the ball:
So, he enquired who she was, and got introduced, and asked her for the two next. Then, the two third he danced with Miss King, and the two fourth with Maria Lucas, and the two fifth with Jane again, and the two sixth with Lizzy, and the Boulanger --
Moving on to the "might have danced" category:
1. The Scotch reel. Austen actually mentions the reel in Pride and Prejudice, in Mr. Darcy's exchange with Elizabeth Bennet:
"Do not you feel a great inclination, Miss Bennet, to seize such an opportunity of dancing a reel?"
She smiled, but made no answer. He repeated the question, with some surprise at her silence.
"Oh!" said she, "I heard you before; but I could not immediately determine what to say in reply. You wanted me, I know, to say "Yes,'' that you might have the pleasure of despising my taste; but I always delight in overthrowing those kind of schemes, and cheating a person of their premeditated contempt. I have therefore made up my mind to tell you that I do not want to dance a reel at all—and now despise me if you dare."
It's hard to say whether this exchange implies that Austen herself thought dancing a reel warranted despising someone's taste, or whether she was just making a point of Darcy's snobbery or Elizabeth's perception of it. But reels were danced at the highest levels of society all the way into the early 1800s; indeed, dancing masters found it necessary to stipulate that it was rude for some people to decide to dance a reel while others were doing a country dance:
No Lady or Gentleman must, during a Country Dance attempt at Reels, or any other Figures, in the same room. (Thomas Wilson, A Companion to the Ball Room, 1816)
Reels at this time would have taken the form of either three or four people in a line alternating eight bars of heying and eight bars of fancy stepping. While Wilson, a London dancing master, wrote more unusual figures for reels for three, four, five, or six people, it's not clear that they were danced outside his own circle of pupils or outside of London or even at all.
2. The minuet. This was still in existence in Jane Austen's time, in the form of a dance for one couple dancing alone. It had previously been considered the proper way to open a ball; Thomas Wilson was still insisting on this point as late as 1816:
The most fashionable and proper dance for the opening of a ball is a Minuet. (A Companion to the Ball Room)
Dance masters, of course, had a stake in keeping the more difficult minuet fashionable, since learning it required more of their services than learning country dancing. It's not clear whether people were still going along with this in the 1790s, let alone the 1810s, but it's not unreasonable to think that at some point in her youth Austen might have danced a minuet. Likewise, the allemande, another baroque couple dance involving complex intertwining of the arms, was in existence in Austen's lifetime, though it's not clear that it was ever particularly popular among her class of English country gentry and she neither mentions dancing it nor uses it in her novels.
3. Sir Roger de Coverley. This particular sequence of figures known as "Sir Roger de Coverley, or the Finishing Dance" date to the late 18th century in England, which is Austen's era. She never mentions the dance by name, however, as she does the Boulanger, so it's a mild but not impossible stretch. I have discussed this dance at greater length with details on how to dance it here.
4. Quadrilles. We know that Jane was sent quadrilles by her niece Fanny from a letter of February 20, 1816:
Much obliged for the quadrilles, which I am grown to think pretty enough, though of course they are very inferior to the cotillions of my own day.
It's not entirely clear whether she was sent quadrille music (most of the "new sets of quadrilles" published were simply new music for the first set of quadrilles) or both music and dance instructions. It's also hard to say whether her opinion was informed by actually dancing them or simply looking them over. She does not mention dancing them herself, and to an experienced dancer of cotillions the figures would been easy to interpret from written instructions. But it is possible that in the last years of her life, she did in fact dance a set of quadrilles. At that time, it would have been the first set of French quadrilles: Pantalon, Été, Poule, Trenise, Finale. New quadrilles such as the Lancers and Caledonians were still in the future.
5. Much less likely but distantly possible is a special case of a reel: the nine-person dance known variously as the Bounky, Bumpkin, or Country Bumpkin. I have four sources for this dance, three of them Scottish and one unknown (title page missing), ranging from the late 18th century (hard to date without a title page, but typical of the 1780s or 1790s) to dated sources of 1805 and 1817 to another undated one which I would put at c1830 judging from the illustrations and the dance mix. We have no evidence whatsoever that Jane Austen danced the Bumpkin -- and it is distinctive enough that I feel she would have mentioned it had she done so -- but it is certainly documentable to her lifetime.
And then there are things Jane Austen didn't dance, regardless of how often someone thinks it's a good idea to put them in an Austen movie:
1. The waltz. The waltz as a couple dance (as opposed to waltz music, which was already well known) came to England in the early 1810s, towards the end of Austen's lifetime. It was considered quite scandalous, since the couple actually "embraced" on the dance floor rather than courteously limiting touch to hands. It was introduced from the top of society downwards and from major urban centers outwards. Austen was not of top-flight society; even the wealthier characters in her novels are not earls or dukes. She did not go to Almack's, the exclusive social club of upper-class London society where foreign dignitaries helped introduce the dance, or travel on the Continent, where the waltz was established much earlier on. And by the time the waltz came to England, she was a spinster in her late thirties.
Her characters do not waltz either; the one time waltz music is mentioned in one of her novels (Emma) it is in the context of using it for a country dance, which had been the practice in England since at least the early 1800s:
Mrs. Weston, capital in her country-dances, was seated, and beginning an irresistible waltz; and Frank Churchill, coming up with most becoming gallantry to Emma, had secured her hand, and led her up to the top.
While waiting till the other young people could pair themselves off, Emma found time, in spite of the compliments she was receiving on her voice and her taste, to look about, and see what became of Mr. Knightley.
The reference to leading her "up to the top" means up to the top of a set of dancers: the two were to lead off a country dance, as shown by the waiting until the other dancers could pair off and join the set.
So it is very unlikely that Austen ever learned to waltz or would have been anything other than scandalized by it. Likewise, she would not have known the sauteuse, a duple-time leaping waltz which was also known at least in major urban centers by the mid-1810s but does not seem to have spread further at that time.
2. Dances from Playford manuals. Late eighteenth and early nineteenth century people were fashionable and of the moment, culturally, paying attention to what was new and up-to-date. Collections of new country dance music (consisting of hundreds of new tunes) were published on an annual basis during Austen's lifetime, and the style of country dance music, figures, and steps had changed substantially from the late 17th and early 18th century. Austen was no more likely to dance a 75- or 100-year old dance than she was to wear fashions from a hundred years earlier. So, despite their constant presence in movies and on CDs that purport to be Regency- or Austen-era, dances like "Hole in the Wall," "Mr. Beveridge's Maggot," "Childgrove," and "Grimstock" (all dating from 1650 to 1710) are nothing Jane Austen or her characters would have been caught dead dancing. They are beautiful dances, but they are the dances of a different era than that of Austen and her novels.
So, as a general rule, if you can find it in a one of the Playford manuals (published between 1650 and 1728), it is not something Austen would have danced. (You can browse Playford dances by title here, if you want to check a particular one.) The exception musically would be Sir Roger de Coverley, whose tune was published in Playford, though with figures completely unlike the "finishing dance" version of the late 18th century. Oddly enough, in 1816 Thomas Wilson also republished the Cushion Dance, a sort of dance game involving kneeling on a cushion and singing, but he offers a new tune and dialogue for it "for the purpose of showing the difference between the manners and customs of Ball Room Dancing a century ago and those of the present time." I've never seen the Cushion Dance mentioned in this era outside of Wilson's one book (it doesn't appear in any of his many others), so I wouldn't regard it as something common to the time period and could easily believe no one ever danced it in this era.
3. Going the other direction temporally: the polka, schottische, mazurka, and galopade all arrived in England after Austen's death. The Virginia Reel is an American version of Sir Roger de Coverley. "The Congress of Vienna," despite its name, is a choreographed waltz written in the 1970s. "Margaret's Waltz" is another late-20th-century choreography. As noted above, sets of quadrilles other than the first set were not yet popular in Austen's lifetime. Contra "buzz swings" were not yet being used, and duple minor or improper country dances had not yet been reinvented after dying out earlier in the 18th century.
For a light-hearted set of quick tips on how to judge dance scenes in Austen or Regency-era movies, please see my previous article, "Real Regency Dancers Don't Turn Single."
For dance music for many of the dances described above (including a full five-figure quadrille, reels, and the Boulanger), I recommend the CD The Regency Ballroom, by Spare Parts, available here.