Anna Shipman : JFDI

How Anonymous Submissions Affected Gender Balance at our Conference

17 November 2013

Mainly inspired by this article, I proposed that we make submissions anonymous for SPA2013. The committee (four of us) discussed what we thought the outcome would be for some time, before deciding to just try it and see how it worked out. And here's the answer.

The conference

Firstly, a bit about SPA. It stands for Software Practice Advancement, and it's a very interactive conference – you are unlikely to be sitting in a talk. You'll be writing code in a new language, or doing a group exercise in a process track, or taking part in a goldfish bowl discussion, or actively learning in another way. It's a great conference, and if you're interested in checking it out we are holding a free one day taster with some of the best sessions from this year in Cambridge on 23rd November. MiniSPA. It's free!

Enough pitching. I've attended SPA three years running and became co-Programme Chair for the 2013 conference, and while I think it's very good, one thing I think would improve it is more women leading sessions. I would also like it if we could get a greater diversity of speakers in other ways, but I thought I would focus on women for starters.

Stats

The first year I attended, SPA2011, there were 28 sessions, with a total of 43 session leaders between them. Of those 43 session leaders, 38 were men and the remaining 5 were women.

Other interesting things I noticed were that four people presented more than one session at the conference – all four were men. Only nine sessions were led by just one person, and of those nine, eight were men.

2011

5/43 = 12% of the session leaders were women.

1/9 = 11% of the solo session leaders were women.

0/4 = No women ran more than one session.

The next year, there were 30 sessions and 42 presenters. Four were women. Again, the five presenters running more than one session were all men. Ten sessions were led by a single presenter – 8 men and 2 women (one was me!).

2012

4/42 = 10% of the session leaders were women.

2/10 = 20% of the solo session leaders were women.

0/5 = No women ran more than one session.

How we did anonymity

At SPA we have three stages after submission of proposals.

  1. A feedback stage, where people offer advice to help the session submitter improve the submission (which can be edited up until the deadline).
  2. A review stage where at least three reviewers grade a submission according to whether they think it should be included in the programme.
  3. A programme meeting, when we pull the reviews together and decide on the programme.

In previous years, the presenters' names appeared alongside their submissions all the way through. This year, we made them anonymous right up until we had put together a draft programme, and then we did the big reveal, just to check that we hadn't scheduled one person to do several sessions at the same time, for example.

Results

So did it make a difference?

At SPA2013, we had 27 sessions with 46 presenters, 40 men and 6 women. Of those, two were repeat session leaders (both men) and 10 sessions had single leaders, of whom one was a woman.

2013

6/46 = 13% of the session leaders were women.

1/10 = 10% of the solo session leaders were women.

0/2 = No women ran more than one session.

So no, not really.

That is – to the number of women speaking. A lot of the conference feedback praised the variety of sessions and topics, more so than in previous years. So we did something right. I also don't know whether it made any difference to the number of women who submitted, as I don't have that data.

But there were two other very interesting outcomes.

Lack of positive bias is good for everyone else

The first unintended – but good! – outcome was that people new to speaking at the conference felt much more able to submit.

A very representative comment from the survey I conducted was:

"Since I haven't spoken at SPA before, I expect being anonymous helped me compete with regulars."

And correspondingly, the reviewers had a lot of very similar comments. Here's one that sums up most of them:

"We used to have too many reviews along the lines of 'a bit weak but I know Alice and Bob and they'll do a good job'. They didn't always do a good job and it resulted in too many repeat speakers, which I think new potential speakers find off-putting."

Many of the reviewers admitted they didn't actually like anonymity but realised it helped prevent bias. Of course, we are generally not aware of our subconscious biases *against* people, but one thing that anonymity made reviewers aware of was our tendency to be biased *towards* people we know. But of course, that kind of positive bias means that someone merely unknown will have to work harder to get their session considered.

Positive discrimination can be off-putting

The second benefit of the anonymity became apparent in a scenario that I hadn't even considered. One submitter proposed a session about probabilistic data structures, which looked excellent, but her language of choice was Haskell, and most of the feedback focused on attempting to get her to change to a more widely used language. Based on that, she concluded that the talk probably wasn't right for SPA, and assumed that she would later hear it had been rejected.

In fact the talk was very popular with the reviewers, receiving top grades from all of them, so it was a shoe-in for the conference. But when we contacted her to let her know she was in the programme, she initially said she was no longer able to give the talk. A day or so later, she got back in touch to say – to our relief – that she actually was available, and explained her initial reluctance:

"Truth is the session feedback concluded that the idea wasn't right for the conference, so I inferred that you must've accepted the session because I'm female, and had a bit of an emotional reaction; like any human I want to have opportunities based on my own merits."

In actual fact, this was not what had happened, as neither I nor any of the reviewers knew who she was. Her session had been accepted purely on the merits of the submission. And a massive advantage of the anonymity we had put in place meant that we could claim with certainty and evidence that we were not making decisions in a biased way, either for or against her submission based on her gender.

A possible objection

Some people surveyed felt that it might be appropriate to remove the anonymity at the review stage. Ability to present content can be at least as important as the content itself, and people felt the best way to get this information was from knowing who the speaker is. Here is a representative comment:

"I think that the knowledge of a presenter's past delivery can be a big differentiator when you get submissions of similar quality."

But really, this is just another way of saying we want to know when people are well-known – we want to put Alice and Bob in because they always do a good session.

Yes, a speaker's delivery and structure makes a huge difference to how well a session works. However, at SPA we assign shepherds to first-time presenters (or anyone who wants one) and they can be very helpful – my excellent shepherd Adam Iley, among other things, arranged a run-through at his workplace to allow me to practice my session with an unfamiliar audience.

So if we do accept a promising session where we don't know that the speaker is good, we might be able to help with that; and conversely, I tend to think that a good presenter should be able to write a good submission so that will shine through.

Next!

In any case, the committee felt that on balance it was positive, so we are continuing with anonymity throughout the whole process this year.

Why not get involved? Submit a session, or get involved in giving feedback on sessions, or both!

Check out the call for proposals, and have a look at the previous programmes I've linked to above for an idea of the kinds of sessions we like.

And if you have any questions, or want to get involved in giving feedback, please do get in touch.

If you’d like to be notified when I publish a new post, and possibly receive very occasional announcements, sign up to my mailing list:

Email Format