Hey Debriefers,

When we do advocacy we’re always asked for “evidence”, but data on disability is hard. We don't have enough, and it's complicated to gather. So we stick to some greatest hits, like saying 15% of the world's population are persons with disabilities.

I wanted to understand more, so I had this conversation with data doyen Dr. Jennifer Madans. She's been at the heart of international collaboration that developed the approaches we use to gather data on disability.

Jennifer shows us where the 15% comes from, what we can use it for, and its limitations too. We set the scene for that by understanding the importance of data on disability to inform policy and how to think about gathering data on groups within a population and prevalence rates.

From there we dive into the Washington Group Questions. This part of the discussion assumes you already know about them: they are the six questions we recommend people to use if they want to know about disability in a census. They don't ask about disability directly, but ask about functioning in six different areas, such as whether you have difficulties walking or climbing stairs.

If today is the day you needed a discussion on some of the finer points of how we use them in different contexts, then you're in luck. If today is not that day, then do stay with us, and scroll down to where we zoom out again, discussing opportunities on data for the disability community.

We close by talking about what more's to be done on data, and what civil society and organizations representing persons with disabilities are doing with data. Disability data is sometimes under more scrutiny than other areas, but the way we are responding to that means there are still important developments on the horizon.

I learned a lot from this conversation, and it's changed the way I understand what the 15% is and the place data has in making the changes we need.

Disability Debrief is a newsletter by me, Peter, that I make keep track of how the world is changing for persons with disabilities. Catch up with the previous edition, the November guide to international news. I love when people get in touch, so do comment, reply or find me on twitter @desibility. And of course sharing this newsletter helps more people nerd out on disability.

Conversation with Jennifer

We spoke on zoom, and here's our conversation edited for clarity.

“I did not realize that this was going to be a lifelong commitment for me”

Peter: Before we get technical, it'd be nice to know from a personal point of view, what excites you about disability data? What drives you to keep getting into it?

Jennifer: I’ll tell you a little bit about how I came to be involved in disability statistics, which was not a very direct route. I've always had a great interest in how evidence should be used to inform public policy. You have an idea of where you want the world to be, and you want to get the world to be that way. But if you don't know how to make that happen, you might actually have the opposite effect than the one you intended. So, you need some kind of evidence. And, through my training and education, I focused on the importance of data, and the information systems that give you the evidence that can inform policy.

I was trained in sociology and population studies. I'm actually a demographer with a focus on survey research and statistics. After graduate school I decided to focus more in the health area. I worked on a variety of topics but then started to focus more on the importance of data and the related measurement challenges.

I became particularly interested in ways to characterize health that were not based on diseases and pathologies. How do you do that?  Trying to answer that question got me interested in functional health, which is related to, but not exactly the same, as disability. I started going to some of the disability meetings and conferences and realized that there were a lot of measurement issues that were really holding progress back so I became interested in how might those measurement issues be addressed.

Then, in 2001, I got a call from the Chief Statistician of the United States, asking if the National Centre for Health Statistics, where I worked, would be willing to host the first meeting of a City Group that was going to be set up to deal with disability statistics.  I said of course we will do that because it's important for our day jobs as health statisticians to collect disability statistics and to do it appropriately. For me, it was very interesting from that measurement perspective, and it had this added benefit of doing something to reduce disparities and deal with people's ability to live a full life.

And, as they say, the rest was history. I did not realize in 2001, or 2002 when the Washington group started, that this was going to be a lifelong commitment for me. I have never regretted that decision. It has been extremely satisfying to me, both intellectually and personally.  I still have the interest in measurement – what are you trying to measure, how do you measure it – but also doing it in this in a much broader context of human rights frameworks.

Most of the time statisticians talk to each other. We deal in our own little world. But here I have been dealing with development ministries, and NGOs, and Organizations of Persons with Disabilities [OPDs], and UN agencies.  I felt I was making a contribution to a much bigger universe. I have the intellectual interest and I have the public welfare interest. I wouldn't give up this work because I get so much personal satisfaction out of it.

Peter: What a great combination: how wonderful to have that in a career, these different intersections.

Jennifer: Extremely lucky. I didn't know what I was getting into but I am very thankful that I had the opportunity.

Peter: That's wonderful. And your answer showed us where the data sits: you need the information systems underneath and then from those, you get the evidence, and then from those, you can really drive policy in the right direction.

“There is no one prevalence rate.”

Peter: Do you feel that the measurement issue is solved now? We’ve got the Washington group questions, we've got extended set, children are hard so we're doing some extra things for children from the measurement point of view…

Jennifer: When you think about disability, unlike some other topic areas, it is totally encompassing. The definition, the social model in the [United Nations] Convention, includes pretty much everything about a person -- all of their characteristics, everything about the environment that they live in, and the interaction of those two domains. That's everything. There is really nothing left out.

Will the measurement issues be solved? Never. It's just too complicated and no measurements issues are ever totally solved.  Do I think that we have made progress on the frameworks for thinking about how to collect data; I do. I think that we have now a better idea of how to approach measurement in a way that provides the necessary information to inform policy, which will vary depending on what you want to use the information for.

But there still is a message that has not really been completely understood. There is no one population with disabilities, there's no one prevalence rate, there is not one nicely defined set of data that you need to inform policy. We need a way to deal with data so that whomever is using the data understands what the data can and can't provide and how the data should and shouldn't be used.

It’s a big ask. I’ve dealt with policymakers and politicians enough to know that they don't want detail and they don't want the complexities. I understand that, that's the world they live in. But, underlying that has to be a greater understanding of the complexity of what we're talking about.

Peter: Even though people want fixed answers, those are the underlying things that mean we’re not going to get fixed answers.

15%: “It is a useful number for some purposes”

Peter: Let’s jump into the most famous number, the 15%, and see how these things come up. Since the 2011 World Report and Disability, every document I write says that that 15% of the world’s population are persons with disabilities. Before that we used 10% but that was kind of made up. How do you understand 15%?

Jennifer: It is a useful number for some purposes, but you do have to understand where it comes from, and how to use it. From a policy and an advocacy point of view, it sends a message that is very useful: disability is not a very rare thing. We're not talking about a very small population that you can ignore. That’s mostly how it’s used, and I think that's serves a purpose. But the issue is so much bigger than that. People think that it has to be that percent, that it is somehow a gold standard, but this can be detrimental. For some uses it is not the right percent.

Peter: Give us an example of a type of use that isn’t the right way to go.

Jennifer: This comes back to what does the word disability mean. It means very different things in very different contexts to different people. For some people in government, when they think about disability, they are thinking about the population that will be eligible for some kind of social protection payment. If they have to give 15% of your population benefits – that presents a problem for them. But, the 15% is probably not the size of the population that would be eligible and the confusion stops the conversation.

On the other hand, if you're thinking about a population that could benefit from Universal Design, for example, in a transportation system, 15% is not the number you want to build for. You want to build for a much bigger number. Prevalence is a byproduct of a way of defining a population. Once you put that definition in place, you can figure out the percent of the population that meets the definition but the real question is how to define the population of interest.

“In order to identify that population, you have to define disability.”

Peter: Prevalence is then related to how you define disability.

Jennifer: That's exactly right. The Convention refers to the rights of persons with disability. In order to identify that population, you have to define disability. Then, you have to figure out how to take that definition and collect information that allows you to identify the population. Once you define that population, you can get a prevalence for that population. That 15% was obtained based on an underlying conception of disability and using a specific database to make the estimate.

In the Convention disability is the interaction of individual functional characteristics and an unaccommodating environment that results in participation restrictions. That's a very complicated definition. It is necessary to unpack the definition when trying to translate it into a data collection tool. There is the added challenge that references to the population with disabilities often focus on the functioning half of [the definition]. You can see that in the Washington Group questions, which ask about people having functional difficulties. The full definition is then addressed by disaggregating other characteristics by disability status. This simplifies the measurement challenge and results in higher quality data but multiple pieces of data are needed to address the full definition.

Focusing on functional difficulties still presents challenges.  Functioning and disability aren’t inherently dichotomies defined by “yes” or “no”. They are continuums and, to make things more complicated, they are composed of many different domains. all of which are continuums.

We still have the requirement to identify this population with disabilities to determine if the requirements of the Convention have been met.  To do so it is necessary to select a point on the continuum that splits the population into those with and without disabilities. I can do that in a variety of ways and the choice of definition will have huge implications for prevalence. Depending on where you put the cut-point you can get a low prevalence or you can get a very high prevalence.

In this sense, disability is not that different from other things that we are more familiar with in terms of needing to define a cut-point on a continuum. We do it with age all the time. The percent of the population that is ‘old’ will depend on where you put the cut-point – some put it at age 30, others at 85.  We have gotten used to these different estimates for age.   We have to get used to it in the area of disability.

Generally, when you make decisions on cut-points, you want to create groups that are very similar in terms of the characteristics of the group members. If I'm going to identify two groups, I'd like those groups to be very similar within their group, but very different from the other group.  If I create groups that are very heterogeneous, the members will have very different functional abilities. They're also probably going to have very different challenges in terms of their environment.

If I create a group, let's say, that includes people who have some difficulty seeing but also includes people who are blind, that group is very heterogeneous in terms of vision. They're going to have very different participation experiences in an unaccommodating environment.  There will also be more people that have lower levels of difficulty and fewer people that are blind. The experiences of people with lower levels of difficulty tend to overwhelm the group averages.  If this group is identified as having a disability it will seem that their participation levels are high even though some people in the group will have much lower participation levels.  Back to the 15%, that number comes from the selection of a particular point on the continuum.  Other, equally reasonable selections would lead to different, sometimes very different, percentages.

Peter: We're going quite into the technical challenges of measuring disability. The other challenge just to flag it, is you could ask people if they think they have a disability, and you get very low response. Most of the 15%, don't respond like that.

Where the 15% comes from

Peter: In this context you've given to the challenges, the 15% number comes from health surveys that were done in 2002, 2004. Part of the reason we get that number is where they chose the cut off. Can you guide us through, what is the 15% number?

Jennifer: The data come from the WHO health surveys that were done during that time period. They did not have information from all parts of the world so had to do some modeling to make an overall estimate. That's fine. The surveys included a lot of different questions related to health but the surveys were not particularly designed around functioning. The analysts took the information available and created a composite measure (a continuum) of functioning using, if I am remembering correctly, a statistical technique called Item Response Theory which makes some assumptions about the nature of the relationship among the measures.

Every case in the data set then got a score based on that model.  A cutoff point to define the population with disability was then selected and, again if I'm remembering correctly, the cut-point they chose was the average score of people who had certain chronic conditions related to functional difficulties. We could discuss whether the underlying models for filling in missing data or for defining the continuum were good models or bad models, whether the data underlying the models were appropriate, whether the assumptions of combining the data were met or whether the selection of the cutoff was reasonable: but in the end the method produced the 15%.

Peter: The takeaways are that this kind of health-driven data, rather than functional-driven data, and it's quite complex modeling to get the result.

Jennifer: Yeah. Would we prefer better underlying questions? Definitely. The modeling is a big component, and some people are more comfortable with complex modeling than others. The quality of the model and reasonableness of the underlying assumptions are often hard to evaluate. And then, there is the question of why that cut-point.  The decision is justified in the report but it is one of many that could have been used and there is no gold standard to guide the decision.  It must be made based on the use that the data will be put to.

Peter: Yeah, it's a choice.

What the 15% tells us

Peter: What do you learn from the resulting 15%? You said it's a good way to remind policy makers that this a broader group then you thought about. How do you take it?

Jennifer: I view it as a rough estimate of the population that has functional difficulty at a level that is of major policy concern in terms of ensuring full inclusion.  But is that number 15% or 17%, or 10%?  That's a level of specificity that we don’t have. We do know that it is not 2% or 50%. Many other sources of data provide estimates in the same ballpark.  The prevalence is less important than the characteristics of the population identified and the use that the information on that population will be put to.

Peter: Are you worried about the age of the surveys that informed it?

Jennifer: Not so much. I worry more about the underlying assumptions of the model. I prefer a less model-based approach so that it is possible to concretely describe the characteristics of the population identified, such as the population having at least a lot of difficulty in at least one of six domains, as opposed to explaining that these are people who have a score of 80, or whatever it was, on this composite measure. Especially when dealing with someone who's actually going to implement a policy or program.

Another point on the size of the prevalence estimate, I know there's sometimes a push to say, “well, the bigger the number, the better,” and that from an advocacy point of view, you want to have a large number.  But this can have unintended effects because of the heterogeneity we discussed. If you identify a large population as having a disability, that population will look more similar to the population that doesn't have a disability. And, when you do the disaggregation, which is our primary source of information about what needs to be done to meet the Convention requirements, the differences are much less. We have shown this with our analysis of data using the Washington group questions.

The differences shown through disaggregation are the largest when the group with disability is the smallest, because that group is very homogeneous and has a lot of difficulties. When you start expanding the group you include people with better levels of functioning. And, as a group, they have fewer challenges and start looking more like the group without.

So while you want to identify the population with disabilities, you really want to go beyond that and start looking at subgroups within the larger population with disabilities. You want to look at, maybe not a dichotomy, but more groups. You want some way of bringing in the continuum more than you can with just a dichotomy. You can do that with the method used for the World Report, you can do it with other methods. I think that is the more important issue, not whether it's 15%.

“If I don't know about the characteristics of that population, then I don't know how to develop policies to increase inclusion.”

Peter: Unpacking different approaches to data, one of the things I'm interested about is in terms of living up to “nothing about us, without us”. We don’t have a badge saying we have a disability or not. If an organization said to you, look, we want to represent the views of this 15-ish percent, how could they reach them?

Jennifer: A traditional way to identify a person with disability was to ask – “Do you have a disability?” This question tends to give you a very low prevalence for a variety of reasons. In many countries there is stigma associated with disability so many people won’t say they have disability. There's also the issue of whether someone identifies as a person with disability. A person can have certain characteristics that would qualify them as having disability, but doesn’t identify as a member of the group. If you're interested in people who self-identify with that label, then you ask the question but still need to be concerned about stigma.

I don't think that's the only thing that people are interested in, I think they're interested in the underlying characteristics that are related to disability as a concept. This leads to asking about actual functional abilities or difficulties that people might have, because that is less stigmatizing and is getting at the core definitional parts of disability as a concept.

This is not a great analogy but illustrates a point: in places where health care isn’t widely available, if you ask if a person has a disease, they will not be able to tell you if they've never been diagnosed with it due to lack of health care. But you can ask about symptoms. And then, you can determine whether it's likely that they actually have that disease based on the symptoms. This is similar to not asking if someone has a disability but asking about their functioning.

Asking directly about having a disability gives some information but doesn’t provide information on functioning that might cause difficulties participating.   The policy issue is changing the environment to meet the functional demands. If I don't know the functional demands, I don't know how to change the environment. Even if I could identify the population with disability by asking if they have a disability and it's the perfect number, if I don't know about the characteristics of that population, in terms of what their functional challenges are in their environment, then I don't know how to develop policies to increase inclusion.

“It's not sufficient to get questions in just one area.”

Peter: You would have to go beyond people that use the label of “disability” to get a broader group of people that experience those situations.

Jennifer:  Yes. This also relates to using the functional approach, as opposed to a diagnostic approach. Questions are often raised about specific conditions such as identify those with autism since the functional questions don't ask about autism. That's absolutely correct but it is possible to get information about conditions in addition to the functional information.

The intent of the Washington Group questions was not to focus so much on the conditions that cause the functional problem, but to talk about the functioning because that's where you're going to make the interaction with the environment work. It isn’t the diagnosis, but how the diagnosis manifest itself in terms of functioning that creates the lack of connection between the person and the environment.

I should also say that it's not sufficient to get questions in just one area; it is necessary to get full information in all of the parts of that overarching disability definition. In 2001 there were a lot of data needs but a lot of poorly collected and confusing data. We were in a position where we really needed to stop doing what we had been doing, because it wasn’t getting us where we wanted to be from a data standpoint.

At the first meeting of the Washington Group, we went around the room asking every country, and there over 70 countries represented, “from a government point of view as representatives of your national statistical offices, what kind of information is the most important for you to get right away? What do you really need right now?”

Although we didn’t use the term at the time, the responses primarily related to data disaggregation. Data were needed to be able to identify the population with disabilities to determine how they were functioning relative to people without disabilities. Data on disparities were needed to feed into policy issues about improving the status of persons with disabilities. The first priority was data from the census because the census is a main data collection mechanism for many countries. The first objective of the Washington Group then was to develop questions that allow for the identification of the population with disabilities for disaggregation and this predated the convention.

Peter: As you told us at the start, the complexities and all-encompassing nature of disability mean that each data approach we take is going to be partial and one crosscut of it. You just described how using the functional difficulties approach is a way in, and then the other part of that is what is the purpose of the survey.

“If the questions can be used in a census, they can be used anywhere”

Peter: Talking about the correct use of the Washington Group questions, is that they're designed for censuses, right? I see organizations that have some target population, have beneficiaries, and they're like, “oh, we want to help disabled people now, identifying disabled people is hard, we'll use the Washington Group questions.” This is quite a common use but isn't exactly what the questions are made for.

Jennifer: The questions were designed so they could be used in a census which has certain requirements due to the way censuses are done. Censuses are huge data collections and are very expensive so there can't be a lot of questions on any topic. And some of the questions that might have been of interest just aren't considered appropriate for a census, given the nature of that data collection. At the beginning, the request was for one question because that's all there was room for on the census. We were pretty much convinced that, if only one question could be asked, don't ask any, because one question just cannot produce good data for the intended purpose. The Washington Group tried to develop the shortest set we could using the most direct and easy to administer questions for use in a census.

The good news is that if the questions can be used in a census, they can pretty much be used anywhere, because the census is one of the most constraining data collection environments. The questions have to be easily understood and be asked by enumerators who don't have a lot of training so if you can put them in a census, you can put them in a survey. Then, the Washington Group started getting questions from many NGOs, INGOs and others who were interested in using the question for program evaluation. Initially we weren't sure about this application, we hadn't really thought about it.

For program evaluation the question is “who are you trying to identify, and for what purpose?” If you're seeing if your program is including people with disabilities, we know that we're missing some domains in the questions used for the census. We don't have a comprehensive measure of psychosocial disability. You couldn't do that in the census. If you use only the short set of questions, you will be missing those people.

If you're doing an evaluation of a programme, you have to determine whether this is a limitation that will affect how you interpret your data. Sometimes it doesn't make that much difference and sometimes it does.  Other questions could be added to address any limitations.

“The use of the questions for eligibility is much more complicated”

Peter: Let's drill down a bit on how a few organizations might be using it. They might be doing some evaluation and then see what was the disability profile in the population they served. But, they might also be using them as eligibility, right? They might be doing a survey of the population to begin with. If you've answered yes to this, then we're going to give you more support.

Jennifer: The use of the questions for eligibility is much more complicated and much more difficult to answer. As noted, the six questions are missing some major domains. While questions can be added to get some of those people, some will still be missed. To the extent that your program is trying to identify every person who might be eligible, that is going to be a limitation. The questions are also not very specific and likely don’t address program eligibility requirements which vary across programs.  Again, there's no perfect data collection, in all of these contexts it is necessary to determine the advantages and the limitations but using the short set is most likely problematic for eligibility determination.

This comes up a lot when you're dealing with the child functioning module and its use in schools. It may be fine if you are doing the data collection to do an evaluation based on disaggregation of how well students with and without disabilities are doing. If you're using it to identify individual children or for some kind of program or accommodation within the school, some children will be missed and some who do not meet requirements will be included.

It may also be necessary to modify how the questions are used.  There is an international cut-off which defines the population using the Washington Group question as having a lot of difficulty in one of the six areas. The standard is for international reporting on disaggregation, such as for the SDGs, so that everybody is making their cut at the same place. We felt that for most policies this was more appropriate than defining disability as having at least 'some difficulty'.  While in a sense, arbitrary, that was the decision that we came to, based on how these data would be used.

That is different than using a set of questions as a screener. For a screener, the objective is to make sure everyone who might be eligible for a benefit is identified. I don't care if too many people are identified, because there'll be a second stage where it can be determined whether, in fact, they're really eligible. You would identify this bigger group to make sure you get everybody. That group is smaller than the total population, but bigger than the group you're actually going to provide services to. The larger group then goes on to another level of evaluation, where those not eligible are identified. The six questions are not really sensitive enough to be used as a screener for eligibility for services.

Peter: They get also based on people's reported status. For eligibility questions this creates challenges, like other forms of eligibility can have. And that second layer of screening with assessment of disability status is a whole other sort of can of worms…

Jennifer: Eligibility determination is really complicated and not something the Washington Group dealt with. Screeners are only appropriate for eligibility evaluation if they would identify all possible eligible for the second layer of assessment.  But when thinking about using screeners, you have to think about the alternative. No screener is perfect, but, otherwise, you have to do an evaluation of everybody and that might not be logistically possible. It goes back to evaluating the tool given the purpose you are using it for.

“How the data will be used and the appropriate cut-off.”

Peter: You mentioned the threshold using the Washington Questions is a “lot of difficulty” for, quote, unquote, “disability”. I think lots of countries use “some difficulty” to make that threshold. You flagged already some of the pros and cons of this, that, well, a bigger number of prevalence is useful but it doesn't get so much of a difference of the population.

Peter: We’re caught in a bit of a bind, because if the number's small, we feel a bit upset, and people won't listen to us. And, the numbers aren't very big sometimes when you do a lot of difficulties. How do you kind of navigate that tension?

Jennifer: We need to go back to how the data will be used and the appropriate cut-off. Since disability exists on a continuum, there is no one population with disabilities. There will always be this tension if the focus is always on identifying one population.

As we have discussed, there are measurement challenges in defining the population.  The Washington Group uses a set of four answer categories which are labeled with words. We did do some testing where, rather than using words, we used numbers such as on a scale of 1 to 100, how much difficulty do you have walking? It was not successful; it was cognitively difficult and resulted in inconsistent responses across respondents. We spent a lot of time developing the four answer categories with the labels of “can’t do at all” / “a lot of difficulty” / “some difficulty” / “no difficulty”. City Groups work in English. It turns out that the translation of those words is not straightforward…

Peter: “some”, or “a lot”…

Jennifer: “A lot” is easier to translate and respondents have a similar understanding of what “a lot” of difficulty entails across cultures. “Some,” on the other hand, is much more variable in terms of how it is understood. Some people feel any difficulty should be reported because they should not have any limitations. Others are a little more stoic and say, well, everyone has some difficulty, and so they don't pick that answer.

We think it is useful to think of the continuum as anchored at two ends, “can’t do at all” and “no difficulty” at each end. Those are understandable terms. What about “a lot” and “some”? You want the “a lot” to be about two-thirds into this continuum. And, “some” to be about one-third.

When translating you should pick words that place the response categories in the correct place. We have a couple of examples where “a lot” was translated as “severe” and “some” was translated as “moderate” which if you think about the continuum, this moves everything to the “cannot” end. If the “a lot” category is translated as “severe”, it will identify a much smaller proportion of the population as having disabilities than was intended.

So, we spend a lot of time telling users that they have to spend time worrying about how they translate those words. We started out thinking that we should have five categories but that turned out to be harder. It was harder to find words for the five than it was for the four.

Peter: I love this, whichever avenue you go down, there's such complexity underneath the decisions that have been taken.

Opportunities for the disability community

Peter We’ve really focused on prevalence data, right. We get very locked into the obsession about what's the overall number. Are there other statistical tools that you feel are underused by the disability community?

Jennifer: It's not the statistical tool as much as how to use the data themselves. If you just wanted the number, you would ask a set of questions that allow you to put people in different groups, and you get a prevalence.  Okay, now we have the prevalence and the two groups which allows the disaggregation of indicators by those two groups. The indicators can be the SDGs or the Incheon Indicators or any set of indicators of interest in every sector of life: poverty, employment, education, voting, transportation.  When disaggregated by disability you can tell whether you've reached the Convention’s objective of full inclusion.

Assume we have the disaggregation, it is now important go beyond that and there is a lot more information available from the questions used to identify the population with disabilities. We know the level of functioning in each of a range of domains and this information can make a difference for policy formation.

We often get questions about the deafblind community. It is possible to look at that community using the Washington Group questions.  Information is available on severity in each domain which can be combined to identify those with hearing and seeing difficulties at different levels of severity.  The six questions were also the bare minimum, there's also the extended set and the Child Functioning Module which provides additional information.

The Washington Group question obtain data on the functioning side of the interaction; everything else that is needed for the disaggregation comes from somewhere else. The employment experts obtain information on employment, the education experts on education and so on. The grand leap is to put the two pieces together to construct the disparities not only by disability but by the intersection of disability and other characteristics such as gender or migration status.

Once the disparity is identified, what needs to be done to address it?  That requires information on barriers and facilitators, the environmental side. Obtaining information on the environment is also quite complicated, because environments vary a lot. If you ask an individual about their transportation environment, it's not one thing, it's lots of things. The ILO and the Washington Group developed a module that is specifically looking at barriers and facilitators for employment. We have the beginnings of an education module that's looking at various facilitators for kids in schools. This has to be done more broadly.

Much of the data used for policy comes from big data collections which are representative of the population if they're done correctly. But, there are a whole range of other data sources, such as administrative data, that can be used and greater use of these data is beginning.  When using administrative data sources you have to be careful because they only cover those who meet eligibility criteria but not all those who are eligible will be in the system because they didn't apply. There are ways of using that information to further inform the major policy questions keeping these limitations in mind.

Two kinds of administrative systems

Peter: The administrative system is when someone's got an identification with a disability card or receiving a disability benefit?

Jennifer: There are two kinds of administrative systems. There are administrative systems that are not related to disability and cover a broader population. All children in school are in the Education Management Information System, for example. If it was possible to identify children with disabilities within that system it would be possible to disaggregated indicators obtained from the system.

There are other administrative systems that are targeted for people with disabilities and depend on people applying to the program and meeting eligibility criteria.  It is ideal to link administrative data with a broader population data system.  The administrative data provides information on those who have applied and met the eligibility requirements. If the system also obtains information using the general functioning questions used in representative data collections then it is possible to see how many people in that system are similar to people identified, for example, in the census but who are not in that system. Cross-walking across data systems really increases analytic capability, which then increases the amount of data you have for policy.

Peter: I think you've described a lot of things that in most countries, we're not doing. Don't just do a kind of demography of disabled people from the census, do an analysis in the different areas of life. And, the other disability-related data that we need to be gathering, particularly about the environment, and then that, I imagine, needs another 20 years of work to figure that out.

Work by governments and NGOs with other types of data

Peter: Then, you've brought up these links with different types of data, right? How do you link your survey data with your government way of identifying its population? Or, how do you link it with health data or mortality data or all kinds of... or, indeed, just disability benefits, how they link and all of those things? There’s a lot of opportunities left on the table. Some of those are quite hard things, but the first set were things that you could just do with data you've already gathered.

Jennifer: There's been a lot of thought given to some of those issues. There are some countries that are really supporting efforts to harmonize among their administrative systems and between their administrative and statistical systems. No, it's not going to be easy, but there's at least a start at looking at harmonization.

You also have a lot of work being done by NGOs in looking at some of these more complicated issues related to incorporating more data collection on disability in terms of not only evaluating their programs, but also in developing them.

Organizations of Persons with Disabilities and data

There is also a lot of interest among OPDs in doing their own data collection often from members of their organizations. While the populations of members of the organization do not include all people with disabilities, it is a very unique and interesting population. How are they viewing the world? What do they view as needs? How do they see these things fit together? One should make the most out of that available data source and try to combine it with the others.

Peter: So an OPD would both want to understand better views of its members and use data tools to do that, and those of its nonmembers.

Jennifer: Many older people who have functional problems and would be identified by our definition of having a disability don't always themselves identify as having disability and are not involved in OPDs. The population with disability and the membership of OPDs don't necessarily completely overlap which needs to be taken into consideration when interpreting data. But, in terms of getting more detailed granular information about how does that interaction between the environment and having  a functional difficulty operates, well, OPDs are a very good source of information about that.

As we discussed, it is harder to get data on the interaction because environments are so different and difficult to measure. How do I ask a question on a survey that really leads me to understand why somebody is having trouble with employment? It is because there isn’t accessibility in the transportation system or a building isn’t accessible or both? This is very complicated to do in a standardized way.

That's why disability, when you think of it appropriately, is so encompassing. There's data in all kinds of places that can be brought to the table and used, as long as there is an understanding of how to appropriately use those data, and they aren’t used for something where they will lead you down the wrong path. There is a real problem when a decision is based on information that's not going to get you where you want to go.

Peter: You’ve shown us the surface of the complexities and an overview of different approaches that can be taken.

Demands made of disability data

Peter: We’re often on the defensive in data. People will say to us, look, you don't have enough data, you need evidence-based, and we suddenly feel we're not wearing any clothes. And, we don't have the same statistics that economists wave about in different areas. But, you've shown us the different things we could do with data. We have all different approaches.

Jennifer: You also have to understand that if you really looked at those data the economists are waving around you would see a holes in it. This is a pet peeve of mine. For some reason, people are requiring a level of quality for disability data that they don't require for other statistical topics.

Think of  an unemployment rate. There are many know problems with it. Does it really get you what you want? Who's in the labor force? Who's not in the labor force? How are you asking the question? How do people respond that have different kinds of complicated job situations or work in nonstandard settings? Is it really the employment rate that is of interest? You could do a treatise on why you shouldn't be using some of these data.

I'm not saying that my colleagues in economics are doing anything bad. It's just that we have learned to use the data appropriately to address those questions where the information obtained is fit for purpose even though they are not perfect. You may even get the unemployment rate wrong by several percentage points here and there, and it may not track exactly with measurement objectives, but we know how to use it. And, I think we have to start using that criteria for disability data.

Peter: Disability data, you feel has more scrutiny or more demands of it?

Jennifer: I do. There is a lot of concern about definitions and who is left out.  There is a lot of discussion of limitations which is at it should be. But, the better question is how are the data going to used and to then evaluate the advantages of using the information in that way. Does what the data provide outweigh the limitations and how then can I compensate for the limitations?

Peter: Yeah, that's a really good guide. The high standards you’ve got on data collection are accompanied by a real-world pragmatism. And the standards not necessarily to get something more pure, but to get something more comparable.

Jennifer: Or, if getting the thing that are so pure will be so burdensome that no one will do it. And then, you have nothing.  Even if the data don’t give you everything you need, what are the alternatives? If it's so bad you'd rather have nothing then the data shouldn’t be used.  Things are usually not so clear cut so it is usually possible to use the data in an appropriate way and then try to make them better, or find another way to fill in the gap.

“The next 10 years will be quite interesting and fruitful”

Peter: Thank you so much for giving time today. Do you have any closing reflections?

Jennifer: I think we should end on a positive note. It's a fascinating area from a technical point of view. It's hugely important from a public policy point of view and a human rights point of view. A lot has been accomplished, I think, in the last 20 years. A lot more needs to be done. While it may seem overwhelming at times, if we just approach the challenges in a consistent way and keep moving forward, I think that the next 10 years will be quite interesting and fruitful. There will be a lot of evolving opportunities for data collection that we're not using right now, like crowdsourcing or big data, that can be taken advantage of.

Again, my personal take home is, always understand the process by which the data were collected in terms of how they will be used. It doesn't have to be perfect; it has to be fit for use. Don't discard data just because it's not fit for every use. This requires a certain amount of data sophistication that isn’t as widespread as we would like but, hopefully, we are making progress. There are a lot of opportunities in the future and we can use current data even if they're not perfect as we make improvements.

Thank you for taking time to talk with me.  It has been a real pleasure.

Further resources

The latest big news in data is UNICEF's report on nearly 240 million children with disabilities around the world, the most comprehensive statistical analysis of this so far (November)

The Washington Group on Disability Statistics has a recently refurbished website with a range of resources on the questions and how to use them.

In the international system there is a Disability Data Advocacy Working Group, which also has a useful email list to stay up-to-date with data.

Acknowledgements

Thanks to Jennifer for guiding us through such complex issues. And also special thanks to Elizabeth Lockwood who suggested Jennifer for this interview and helped to formulate the questions.

Until next time!

Peter