In late 2009 ARBP commissioned a survey to poll REALbasic users about the REAL Software beta program.  Here are the results.

1. Are you a member of the REALbasic beta testing program?
ARBP - Just about a dead heat here. 51% yes, 48% are not.
Non-ARBP - Very nearly an even split with just a slightly larger portion being members of the beta than not.

The distribution of yes and no responses is very nearly the same between ARBP and non-ARBP respondents.

Question #2 was asked only of those who are not members of the beta and was
2. You indicated you are not part of the beta program.  Why not?

ARBP - Responses ranged from not knowing how to get to be a part of it, or what's involved to not having an up to date license. By far the most common rely was along the lines of "I had not yet heard about it, actually."

Visibility of the beta program seems to be sorely lacking among ARBP members.

NON-ARBP - most responses were "I'm a beginner"

3. What do you feel are the major benefits of membership?

ARBP - The top response was having early access to the latest versions of RB for testing a user's own code.  Closely behind that was making REALbasic a better product.  A very close third was to be aware of new bugs in new versions and knowing how to work around them ahead of the release date for the new version.

These three items were separated by only 4 responses from the highest to lowest.

From these three responses we can see that knowing what's coming is a top reason to be part of the beta program.  Also having early feedback to REAL to assist REAL make REALbasic better seems to be a real driving concern.

One thing about this question is that people were allowed to select as many replies as they felt they needed to.  From the tight race for top spot it would seem that many people selected those three as their top reasons for participating.

NON-ARBP - Nearly identical results in the non-ARBP member results.
Most felt the big advantage was access to the latest versions, making REALbasic better, and early feedback about bugs to REAL.  This very nearly mimics the ARBP results

4. How often do you participate in the beta program?

ARBP - Of the respondents who participate in beta's 51% indicated they participated in most or all releases.

The nearly even split between those who participated frequently and those who did not is quite curious.

I'm not sure what to make of this at all.

Non-ARBP - 70% of respondents participated every release or most releases.  This is a slightly higher percentage than ARBP members.

5. Have you submitted a bug report?

ARBP and Non-ARBP - Nearly every one who participated in the beta program indicated that they had submitted a bug report.  However, the question did not qualify this in any way to ask if the respondent had submitted the report as part of the beta program or not.

6. Are you satisfied with the handling of the bug report?

ARBP and Non-ARBP - Nearly 80% of the people who had submitted a bug report indicated that they were satisfied with how a bug was fixed.  The majority indicated that the bug was fixed eventually - not immediately.

Only a very small percentage have reports where the bug has not been fixed or was not fixed correctly.

7. Have you submitted a feature request?

ARBP and Non-ARBP - Nearly 70% of respondents indicated they had submitted a feature request.

8. Has your feature request been implemented?

ARBP - Of those submitting feature requests only 44% indicated their request had been implemented in the current or next version of REALbasic.  For non ARBP members this number was slightly lower ( 42%)

A small percentage were not implemented and the reasons given were sufficient for the reporter.

ARBP - Nearly 42% of the people submitting feature requests indicated that the feature they requested has not been implemented and feel it should be.  Non ARBP members had a slightly higher percentage (50%) who felt this way

It would seem there is a fair amount of room for improvement in communicating why features are not implemented or in actually implementing them more quickly. Having such a large pool of requesters unsatisfied can't be a good thing.

9. How much do you feel that REAL software listens to its beta testers?

ARBP - This question is nearly a split between the "Always or Almost Always" and other responses.  53% of the responses are positive (12% always and 41% Almost Always).  "Some of the time" and "Almost never" garnered the remainder of the responses.

For non-ARBP members the split is 50% replying "Always" or "Almost Always" and 50% replying "Some of the time".  That's a curious split and may indicate a certain level of dissatisfaction with the beta programs overall effectiveness in garnering feedback from participants.

10. Do you feel that the beta program, in its current form, is improving REALbasic?

This question had 3 responses : Yes, No and I'm not sure.
Yes was overwhelming the leader here with nearly 70% of the responses for ARBP members and nearly 90% for non-ARBP members.  "I'm not sure" was the next most popular answer among ARBP members.

11. Do you feel that REAL Software does enough internal testing of their software before releasing the product to the beta testers?

Among ARBP members the leading response was No with 40% of responses.  However, it was not an overwhelming winner.  Yes was very close behind with 36% and I don't know picking up the remainder.

With non-ARBP members the leading response was yes with 55%.  No was the reminder with 45%

12. Do you think the new Feedback application is an improvement in the testing/reporting process?

The responses for this category are quite positive. Nearly 55% of respondents felt it was an improvement.

There are still a significant group that either have not used it or feel that, at the time of the survey, felt it was too early to tell for sure.

Responses from non-ARBP members mirrored this distribution of responses.

13. Do you think that beta program participants (who regularly participate) should be compensated?  Compensation could be discounts, RB apparel, community recognition and any number of other things.

Among ARBP members this one is a dead heat between yes (50%) and no (49%)
Additional comments about compensating beta testers seemed to all into the camp of "We do it to help make the product better" or "Compensation incents people in the wrong way".

Among non-ARBP members no was the majority with 66% of the responses

14. What do you feel needs improvement in the beta program?

The most common response is "Release notes" and how they are organized.
Not being able to tell what has changed from one beta to the next is a common theme.

15. What do you feel works well in the beta program?

Perhaps the most common theme expressed is that people like that The REAL engineer's are easily accessible and actively respond on the Beta list.