Sloan Foundation Proposal Round 2

1.) What is the main issue, problem, or subject and why is it important?

Created by Scientists: The web was created by a scientist to accelerate science. But nearly 30 years on, only a small minority uses the web for more than publishing and distribution. The web has fundamentally altered the nature of commerce, education, culture, and civil society through open communication, access to information, and generative innovation at a global scale. But with a few exceptions, most scientific computing is still done by individual scientists crunching numbers offline.

More Effective, but Not Transformed: Software Carpentry set out to help scientists use computing and the web more effectively in their research. We developed curriculum and ran workshops that taught basic skills such as version control, testing, and program design. Our approach has proven effective: participants become more productive, confident in their ability to tackle new challenges, and gain awareness of what is possible. However, they still return to their labs to write their own code, for their own data, by themselves. The web remains a platform to discuss and disseminate results, but not a tool to conduct, strengthen, and expand their work.

The Methodology of the Web: The web is a technology stack: mark-up languages, hyperlinks, and network protocols. But the web is also a set of cultural norms and practices. Its architecture makes it an open lab where anyone can 'crack the hood' and see how something was done. This has bred a culture that sees individual accomplishment as result to be celebrated, but also a model to be emulated, material to be repurposed, and a contribution to a shared endeavor. In this way, webmaking and scientific practice have much in common. So why have scientists not embraced the web?

Three New Blockers: The first issue is skill. In this, Software Carpentry is correct and proving effective. But there are three other challenges:

Technology: Tools for doing science on the web do exist, but most require a sophistication that puts them out of the reach of most working scientists.

Practice: The value of working in the open — APIs, sharing reusable code, mass collaboration, distributed computing — can seem in conflict with established norms around priority, publishing, and attribution.

Scale: A shortage of working models, knowledgeable instructors, and on-ramps to engagement inhibit growth. What exemplars exist have not coalesced to attract converts.

Science that Works Like the Web: We believe knowing how to code — how to create content and work on the web — is a core literacy akin to reading or mathematics. But more than that, we view the web as a 21st century tool, mindset, and force transforming the nature of professional practice. Working not just on, but like the web can accelerate scientific research: data sets become live streams rather than static files; small ideas grow to become collaborative frameworks; and researchers gain access to thousands of participants willing to report bird sightings, analyze old ships' logs, or classify galaxies. Science pioneered open collaboration and the gift economy, but the web has perfected it. We will build from Software Carpentry to re-establish this generative interplay.

2.) What is the major related work in this field?

Two fields of practice relate to our approach: delivering computing and webmaking skills to non-programmers (including scientists), and conducting science on the web.

Teaching Scientists to Code: Several studies (see bibliography) of how scientists use computers and the web, including the largest by Dr. Wilson in 2008-09, have found that most scientists learn what they know about these subjects through osmosis and word of mouth. Most training meant to address this issue does not target scientists' specific needs, places too much emphasis on programming and number crunching, and/or jumps to advanced topics before scientists have mastered the basics. Software Carpentry is the most significant effort to date to address the issue. Since its inception in 1997, the program has taught several thousand researchers the concepts, skills, and tools they need to get more done in less time and with less pain.

Two other projects of note are the Next-Generation Sequencing (NGS) summer school run by Prof. C. Titus Brown at Michigan State University, and the training and seminars offered by the Software Sustainability Institute (SSI) at the University of Edinburgh. Brown, who has also taught Software Carpentry, starts with the same material and continues into bioinformatics-specific topics. The SSI, which was established to train scientists how to write and collaborate around reusable software, has been working closely with Software Carpentry since January 2012.

Beyond science, Mozilla runs complementary 'open labs' that work to build the skills and transform the practice of professions such as journalism (Open|News) and filmmaking (Living Docs). Two years in, these programs are meeting with significant success. The community behind the Living Docs lab has produced Popcorn, a technology that lets video files control web content and vice versa. Wired and Fast Company both called Popcorn the future of online video. Living Docs has attracted partnership and investment from the Tribeca Film Institute, ITVS, and the Ford Foundation. Popcorn is also driving change within industry. Fox Sports and ABC have commissioned Popcorn-based content, and a start-up just completed a $2M venture capital round to launch a business anchored on the technology. Countless hackjams bring together filmmakers, designers, and developers at almost every major film festival. Open|News has achieved similar results, with partners such as the New York Times, BBC, Al Jazeera, and ProPublica working on the web to produce new tools, including HyperAudio Pad that lets audio files be remixed by editing written transcripts, and DocumentCloud, which embeds source materials into digital news stories (more projects at mozillaopennews.org/code.html). Finally, the labs benefit from and work with each other, as evidenced by PBS NewsHour's plans to use Popcorn for their digital coverage of the 2012 elections.

Conducting Science on the Web: Mozilla's efforts are part of an emerging movement to bring science to the web. From pan-science approaches such as the Panton Principles and calls to publish scientific data as open data, to field-specific initiatives such as Blue Obelisk in chemistry, scientists are taking tentative steps towards working on the web. More progress has been made on projects that benefit from engagement with citizen scientists. In addition to the renowned SETI@Home project, Galaxy Zoo has led to the discovery of new classes of galaxies and yielded a wealth of data. The web-based platform built for the project has expanded to support everything from studying climate change using data from ships' logs to categorizing the speech patterns of killer whales. Another related venture is the emerging discipline of data science: ways of manipulating, mixing, and visualizing the enormous volumes of open data. Projects such as D3 and Wrangler (both built by Prof. Jeff Heer's group at Stanford) are using web-native technologies to make high-quality, hackable data visualizations. These tools can be used to conduct advanced research, but also provide accessible and easily understood on-ramps to help people realize the value of open data, visualizations, and web-enabled science.

3.) Why is the proposer qualified to address the issue or subject for which funds are being sought?

Mozilla Foundation: Mozilla is a non-profit organization best known as the makers of Firefox. At the core of our work are programs that move people from using the web to making the web: from consumption to creation. This involves building tools and software, but also helping people learn about the open ethos and building blocks that make up the web. We run a number of programs related to the work detailed in this proposal:

Software Carpentry: Our ongoing project with Sloan to build the technological skill set of scientists and graduate students, detailed in section 8.

Webmaker: The umbrella program for our work related to learning (including Software Carpentry), Webmaker is a set of tools, curriculum and assessments, and communities of practice dedicated to teaching millions of people how to code, shape, and remix content for the web. Webmaker's design is underpinned by significant research into the skills taxonomy behind digital literacy; new and peer-based models of instruction, assessment, and accreditation (Open Badges); and a growing pool of partners including the New York Times, NASA, Department of Education, the National Science Foundation, and the Tribeca Film Institute.

Open|News, Living Docs & Hive NYC: Parallel programs that, like the work detailed in this proposal, build content, tools, and partnerships for and with participants from specific communities — in these instances journalists, filmmakers, and youth.

We also run the Mozilla Developer Network, the largest repository of resources on how to build content for the web; the Mozilla Reps program, comprised of hundreds of volunteers who host events around the world; Mozilla Ignite, a project with the White House and National Science Foundation to drive the development of gigabit-per-second networks; and we continue to engage in top-level strategy conversations that shape the future of the web. Each of these resources will support our emerging work with the scientific community.

Dr. Greg Wilson: Greg will lead the work on Software Carpentry. A 25-year veteran of the software industry, Greg was co-winner of the Jolt Award for Best General Technical Book in 2008 and received ComputerWorld Canada's 'IT Educator of the Year' award in 2010. He started Software Carpentry as a training course at Los Alamos National Laboratory in 1998. Greg holds a PhD in Computer Science from the University of Edinburgh.

Erin Knight: Erin, Mozilla's Senior Director Learning, leads Mozilla's work in the field, which includes building the content and community needed to create a generation of webmakers. Previously, Erin served as the Research Director at the Center for Next Generation Teaching and Learning. Erin holds an MA in Information Systems and Management from the University of California, Berkeley.

Dr. David Ascher: David's role at Mozilla focuses on the exploration and development of new product initiatives within the Mozilla Labs group. David's previous experiences include academic research in cognitive science, as well as leading the development of Thunderbird.

4.) What is the approach being taken?

This project will see us engage in a three-phase approach.

1. Expanding Software Carpentry to teach more scientists how to work on the web;

2. Building an open Webmaking Science Lab, led by a vibrant and respected scientist, to drive the development of tools, practices, and community around science on the web; and

3. Challenging the lab to engage scientists, developers, and designers to build tools and learning content (products) that address research needs and provide paths to broader participation.

Phases I and II are covered by this proposal, with prototyping funding for Phase III. Should Phase I and II launch successfully, we would work with Sloan to engage additional private and public funders, including the NSF, JISC, and Moore.

1. Deliver More Training to More Scientists: Software Carpentry is working. Independent assessment of the pilot phase found that scientists enjoyed the training and that it improved their skills (see question 8). Workshops are almost always over-subscribed and alumni are beginning to run follow-on tutorials on their own. Given this success, we want to increase our ability to reach more participants, expand the training to teach scientists how to use the web in their research, and provide for longer, ongoing support. In this way, SWC will improve the research practices of individual scientists and build the constituency of individuals able to contribute to the advancement of science on the web.

Our Approach:

• Improve and Increase the Number of Workshops: We will double the number of scientists we train each of the next two years (to 2,500). We will also update the workshop content to incorporate recent developments and feedback from the evaluations. These updates will expose scientists to newer, web-based or -aligned solutions in areas including source control systems and databases, and will shift to allow for hands-on experiences through tools like IPython Notebook.
• Expand the Diversity of Participants: In keeping with Mozilla's broader engagement policies, we will work to encourage participation in the workshops by individuals of diverse gender and ethnic identities.
• Include More Content on the Web: We will create and pilot a second workshop to teach scientists what it means to work with, on, and like the web. We will not set out to teach scientists how to produce socket-level code. We will, however, seek to empower scientists with an understanding of how the web works, which will open new possibilities and encourage new projects. The web is not just a set of technologies, but a culture and practice of 'working in the open'. Working like the web allows for transparency, accessibility, 'remixability', and shared contribution to become embedded in the workflow, culture, and tools underpinning scientific research. The workshops will cover these skills, as well as open web technologies, standards and practices, including concepts such as general web and Internet mechanics (HTTP and RESTful practices), APIs (how to use them and how to design them for others' use), data exchange formats (how to publish data in findable ways), ownership (open licenses, copyright, and attribution), and online collaboration (how to find peers, share ideas and participate effectively in open communities of practice).
• Support Peer-to-Peer Mentoring: To support scientists after and between workshops, we will build on our experience with web-based tutoring to support ad hoc peer-to-peer mentoring. This will include screen sharing and application co-piloting, matchmaking services to pair learners with more experienced colleagues, and live, hack jam-style events at and around major conferences.

How We'll Do It:

• Host Pays Expenses: We will shift to financial model where the workshop host covers the costs of the event (instructors will still not be paid for their time, but all travel costs will be covered by hosts). We are using this model for upcoming workshops.
• Recruit and Train More Instructors: We will develop materials and run training sessions to increase the number of instructors able to deliver core material. We will also offer the support and mentorship of more experienced instructors. New instructors will largely be recruited from participants in prior workshops.
• Webmaking Workshop: We will develop a second workshop curriculum dedicated to webmaking skills. The first incarnation will be a single day focused on the open workflow, technologies, standards, and practices mentioned above. We will iterate based on feedback and external evaluations to expand the workshop to a full two days.
• Institutional Engagement: We will seek to have completion of Software Carpentry accepted toward the NSF's requirements for training in Responsible Conduct of Research (RCR). We will reach out to professional societies directly and through their conferences. We will pilot the 'driver's exam' we are developing for the DiRAC supercomputing facility in Europe. We will work with innovative publishers (e.g. PLoS) to create a checklist of computational practices used in the production of papers, creating an easy way to tell if the authors know what they are doing. Finally, we will support an independent, peer-reviewed assessment (longitudinal survey of workshop participants over two years) of the utility of what we teach and the efficacy with which we teach it.

2. Launch a Webmaking Science Lab: The subject area we are planning to tackle — science and the web — is huge and complex. Our understanding of how science could use the web, what learning is useful, and the tools needed to support their work is going to evolve quickly. What we need is a process — an engine — to drive continued innovation, produce tools, explore new practices, and grow the number of scientists, developers, and designers collaborating over the long term.

Our Approach:

• Webmaking Science Lab: We will work with Sloan and our emerging partners (see below) to launch a Webmaking Science Lab: a body of scientists, developers, and designers building tools and content that supports the practice of science (and citizen science) on the web. Mozilla's innovation strategy is founded on these open, community-driven labs. Labs are online (social media, IRC, online training, screensharing, etc.) and physical (workshops like Software Carpentry, hackjams, and festivals) spaces that gather a diverse community of domain specialists, developers, organizational partners, designers, and hobbyists. Participants develop ideas, build prototypes, test iterative builds, and commit further effort to areas showing results throughout a series of integrated, cumulative activities. Once a promising direction is identified, we invest additional staff time to help define a product vision, engage participants in requirements gathering, and push toward the launch of new tools, code libraries, or other widgets.
  SWC provides a baseline of skill and confidence that empowers scientists to understand and engage with members of the broader web community. The activities within the Lab, in turn, provide additional learning opportunities around computational thinking while generating new tools and projects. This cyclical process of continuous improvement, diverse participation, and rapid testing is better at shipping useful products than closed production and lengthy deployment schedules. The Webmaking Science Lab will tackle issues such as building a shared vocabulary and toolset for data visualization (data aggregation, interactive presentations, and to facilitate collaborative and derivative work), real-time collaboration tools that cater to scientists' specific needs (federated identity systems, data ownership), and explore the 'holy grail' that is the hackable scientific paper.

How We'll Do It:

• Recruit the Lab Director: The success of an Open Lab is a function of many things, but starts with the Lab Director. The Lab Director serves as visionary, product manager, and camp counselor, building the community and anchoring the product vision. The Lab Director also serves as Mozilla's ambassador to the target discipline, in this case science, asking how we can engage and support the activities already underway in the space. We recruit an individual who has a passion for the web, but deep credibility in their field. For Living Docs, we engaged an award-winning filmmaker who produced the defining documentary on remix culture and launched the first open source filmmaking project. For Open|News, we recruited a charismatic and respected 'zine publisher, journalism professor, Knight Fellow, and Colbert Report guest with a gift for digital storytelling.
  The Webmaking Science Lab Director will be chosen for their ability to communicate their vision (and the vision of other like-minded people) to the world, particularly to research scientists and professional societies; make connections between partner organizations and relevant groups in Mozilla; and drive the development of tools and products that advance scientific practice on the web. The ideal candidate is someone who has an accomplished track record and profile within their discipline, but has chosen (or desires) a career path outside of academia. We have a few individuals in mind for the role, including Michael Nielson (michaelnielsen.org), Carly Strasser (carlystrasser.net), and Arfon Smith (arfon.org), but we will also advertise the role through Mozilla's Jobs page, our social media outlets, and through direct outreach through our emerging advisory board. These channels will allow us to reach thousands of potential applicants. A job description is appended to this proposal.
• Build an Advisory Board: We will leverage Mozilla's brand and our work on Software Carpentry to rally and partner with key organizations, professional societies, and individuals active in the space. We have confirmed the participation of Prof. Titus Brown (Michigan State University), Mr. Shreyas Cholia (Lawrence Berkeley National Laboratory), Prof. Carole Goble (University of Manchester), Prof. Mark Plumbley (Queen Mary University of London), Prof. Marian Petre (Open University), and Prof. Ethan White (Utah State University). We will also approach Cameron Neylon (PLoS) for reinventing scientific publishing; Fernando Perez (UC Berkeley) for creating an electronic lab notebook; Mark Guzdial (Georgia Tech) for instructional practices; and Jeff Heer (Stanford) for data visualization. The exact composition of the Board will be developed in partnership with Sloan and will include representatives from a range of disciplines within the sciences.
• Integrate with Mozilla's Broader Innovation Activities: Simply put: we know how to do this. We are one of the world's most successful open source projects, so we will build from and integrate with Mozilla's existing open innovation practice. Our expertise in this area encompasses technical know-how as well as softer skills, including building processes, techniques for encouraging repeat contributions, identifying leaders in the nascent community, and empowering local leaders to organize events and feed the results back to the community. The Science Lab will have immediate access to our events, including the Mozilla Festival, our extensive developer community, and the tools we are building to teach webmaking. The Lab will also benefit from our experience running related efforts in journalism (Open|News) and filmmaking (Living Docs) through direct engagement between the Program Directors.
• Independent Assessment: The independent assessment will include detailed, real-time feedback on the success of the Lab, which we will use to shape our activities.

3. Design and Ship Products for Science on the Web: Open Labs are community-driven product development environments. Participants collaborate to produce software, tools, learning materials, and flagship projects demonstrating their application. This product development ecosystem is driven by motivated participants from diverse backgrounds, in this instance scientists, developers, designers, hobbyists, students, and the public. Within the scope of this proposal, we will undertake to produce small prototypes and reference implementations to test potential directions and underpin future proposals. Should we be successful at building a community of scientists, developers, and designers around a meaningful product direction, we will work with Sloan to approach other funders to bring resources to the development activities taking place within the Lab.

5.) What will be the output from the project?

Software Carpentry

Deliverables: An expanded curriculum that incorporates webmaking for scientists; training materials and specialized 'train the trainers' workshops for instructors; and a community to support them.

Objectives & Metrics

• Increase the number of scientists receiving training to 2500/year by end of 2014, ensuring a diversity of participation across genders and ethnic groups.
• Increase the number of credentialed instructors for to at least 50 by end of 2014.
• Deliver webmaking training at least four times to at least 100 scientists in 2013; goals for 2014 will be set based on feedback from pilot offerings.
• Institutional acceptance of Software Carpentry credentials through Open Badges, including training recognized by at least two US universities as fulfilling NSF requirements for training in RCR by mid-2013, and 'Driver's license' required by DiRAC supercomputing facility in Europe by mid-2013.
• Directed outreach to professional societies interested in building the capacity of their membership.

Webmaking Science Lab

Deliverables: A community of practice, including Software Carpentry alumni; integration with Mozilla's broader community-driven innovation events, including hackjams and the Mozilla Festival; and ideas, tools, practices, and lessons that emerge from Lab experiments.

Objectives & Metrics

• Growing science-related code repositories with documentation with ~25 ongoing projects of various scale (modeled on the Source web site we are building for journalists [source-dev.mozillalabs.com]).
• Steady growth in the number of active participants in Lab-related discussion and mentoring (goal is to capture 10-15% of SWC participants).
• Steady growth in the number of people contributing to Lab-based software projects (core group of 25 participants supported by a larger community of several hundred).
• Steady increase in the Lab's profile based on mentions in the press and contributions to its software projects.
• Citations in published papers.
• Frequent referrals to the Lab for answers and assistance.
• Frequent requests for new features and extensions.
• A growing corpus of critical independent assessments of both the tools and the teaching.
• Prototypes and reference implementations of potential product directions.

6.) What is the justification for the amount of money being requested?

Program Direction: Mozilla's Sr. Director of Learning, Erin Knight, and Sr. Director of Software, Dr. David Ascher, will serve as the lead executives overseeing the project. Our Director of Communications, Matt Thompson, and Manager of Global Events, Michelle Thorne, will coordinate and lead outreach and communications, as well as integration with Mozilla's broader events and community-driven innovation activities. We will provide for these costs from our existing revenue. Within the requested amount, an external consultant will be engaged to lead assessment, evaluation, and reporting efforts.

Project Delivery: The work on Software Carpentry will continue to be led by Dr. Wilson. He will oversee and co-author upgrades to the curriculum, develop the webmaking workshop, recruit new workshop hosts and instructors, and build the infrastructure (materials, training sessions, peer support networks, etc.) needed to scale Software Carpentry to thousands of participants. Dr. Wilson will also leverage his experience in academia to foster institutional recognition for the importance of proper technology practices in the conduct of scientific research. Greg will be supported by a dedicated Workshop Facilitator. The Facilitator will deliver some workshops, but will also match instructors to venues, deliver online tutorials, facilitate peer-to-peer mentoring, and manage credentialization. Dr. Wilson and the Workshop Facilitator will co-author the guides and other materials required to expand the scope and participation level in Software Carpentry.

For the Webmaking Science Lab, we will engage the leadership of a distinguished and high-profile scientist to lead the design and development of the Lab and its activities. The individual will reach out to the scientific community and engage them in shaping the purpose, scope of activities, and product vision for the Lab. There will also be a small pool of funding to develop prototypes and reference implementations of potential products. We will also work to secure the resources required to augment the Lab with additional development support.

Wage Costs, Benefits & Overhead: The 30% multiplier includes 7.5% overhead and 22.5% wage costs and benefits.

Travel: We have allowed for one trip per month for Software Carpentry staff, and one trip per quarter for the Webmaking Science Lab Director.

7.) What other sources of support does the proposer have in hand or has applied for?

Mozilla Webmaker: Our work with Sloan will fall within our larger Webmaker program. Funding for Webmaker comes from current and prior revenues received as the shareholder of the Mozilla Corporation; funding from organizations including the Ford Foundation, National Science Foundation, and MacArthur Foundation; and Join Mozilla, a broad-based fundraising and community-organizing effort.

Software Carpentry: Since the start of our work on Software Carpentry this January, we have received money to support travel costs for workshop instructors from Enthought Inc. (a geophysics and computational financial consultancy based in Austin and New York City); from the Sound Software research consortium based at Queen Mary, University of London, to support development of an instructor's guide and delivery of a workshop; and Prof. Brown at Michigan State University has received funding for Software Carpentry as part of a larger NSF proposal. Going forward, we have verbal confirmation of support from Lawrence Berkeley National Laboratory to provide both basic and advanced (web-oriented) workshops, and Prof. Carole Goble of the University of Manchester is including support for Software Carpentry in the UK's proposal for the European Union's ELIXIR program.

8.) What is the status and output of current and/or previous Sloan grants?

In January 2012, the Sloan Foundation provided a grant of $124,625 in support of Software Carpentry. Six months later, we have run 20 workshops for over 600 scientists and two independent assessments have shown significant positive impact. We have made substantial progress on institutional engagement and have shown that our approach is scalable: a quarter of workshops have been peer-led, and we now have over a dozen instructors able to deliver those workshops and online tutorials.

What We Planned: Our original proposal for Software Carpentry focused on four metrics:

1. The percentage of course participants that offer or plan to offer in-person workshops and study groups on their own, as well as recommend the course to their peers. 96% of the participants surveyed said that they would recommend the workshop to others and 16 student-led groups have committed to hosting workshops in the coming months.
2. The scope of badge display and associated reputation. As detailed below, we shifted our focus from credentialing learners to credentialing instructors. Most are further along in their careers and find recognition of special accomplishments valuable as they apply for post-doctoral or faculty positions. We expect to be able to report on this shift in the first half of 2013.
3. The number of institutions engaging in our work and the depth of that engagement. Faculty members at Michigan State University and Utah State University are asking their universities to count Software Carpentry as fulfillment of the National Science Foundation's requirements for training in Responsible Conduct of Research (RCR). If this is approved, we expect a significant increase in interest at those institutions, nearly 100% uptake of badges, and emulation at other institutions. In addition, Software Carpentry is working with the Software Sustainability Institute in the UK to formulate a 'driver's exam' for the new DiRAC high-performance computing facility. The aim is to provide formative assessment for scientists who wish to use DiRAC via an hour-long examination that will include written, recorded, and interactive components.
4. The number of and degree to which former students becoming makers. We initially planned to have learners create new examples and lessons for inclusion in our web-based material. What we found was that people who are interested in contributing would much rather do so in person, e.g., by helping out with workshops or online tutorials. As a result, we shifted our focus to recruiting and training the next wave of instructors. 36 people have helped deliver Software Carpentry workshops, half of them independently, and five of those have run online tutorials reaching dozens of people each.

Where We Shifted: We had planned to develop 15 hours or more of new video content during the pilot. However, it became clear by April 2012 that this would not add much value in the eyes of our intended audience. We therefore re-prioritized as follows:

• We provided more travel funding for tutors, many of whom were former participants in Software Carpentry.
• We are creating an instructor's guide to help scientists become effective facilitators.
• We are recording Dr. Wilson presenting key lessons from the course in order to show instructors-in-training how we teach (rather than what we teach).
• We are developing a badge plug-in for WordPress, which will allow our sites (and thousands of other) to display badges awarded by Software Carpentry and other projects.

These adjustments were decided and managed in accordance with Mozilla's agile and iterative approach to program implementation.

How We Did: We evaluated the effectiveness of the Software Carpentry program through several means. The bulk of the work was performed by Dr. Jorge Aranda, who surveyed and interviewed participants, observed a workshop, and analyzed screencasts of participants working through a programming assignment. Dr. Aranda's survey found significant increases in participants' understanding and use of shell commands, version control tools, Python, and testing techniques. Perhaps more importantly, participants reported better proficiency with software tools; greater concern for issues of provenance and code quality; better strategies to approach software development; and new research questions that have become accessible thanks to an increase in participants' software development skills.

Independently, Prof. Julie Libarkin (Michigan State University) performed a more detailed assessment of participants in a workshop held there, which was attended remotely by students from the University of Texas at Austin. 85% of participants reported that they learned what they hoped to learn, 81% changed their computational understanding, and 96% said they would recommend the workshop to others.

References

1. Jeffrey C. Carver, Richard P. Kendall, Susan E. Squires, and Douglass E. Post: "Software Development Environments for Scientific and Engineering Software: A Series of Case Studies." 29th International Conference on Software Engineering, 2007.
2. Jo Erskine Hannay, Hans Petter Langtangen, Carolyn MacLeod, Dietmar Pfahl, Janice Singer, and Greg Wilson: "How Do Scientists Develop and Use Scientific Software?" Second International Workshop on Software Engineering for Computational Science and Engineering, 2009.
3. Prakash Prabhu, Thomas B. Jablin, Arun Raman, Yun Zhang, Jialu Huang, Hanjun Kim, Nick P. Johnson, Feng Liu, Soumyadeep Ghosh, Stephen Beard, Taewook Oh, Matthew Zoufaly, David Walker, and David I. August: "A Survey of the Practice of Computational Science." 24th ACM/IEEE Conference on High Performance Computing, Networking, Storage and Analysis, 2011.
4. James Howison and James D. Herbsleb: "Scientific Software: Production and Collaboration." Computer Support for Cooperative Work, 2011.
5. Judith Segal: "When Software Engineers Met Research Scientists: A Case Study." Empirical Software Engineering, 10(4), 2005.