HTCondor European Workshop returns for ninth year in Orsay, France

The workshop highlights research organizations’ success with the HTCondor Software Suite (HTCSS), challenges, and possible solutions. Planning for 2024’s workshop is already underway.

The ninth 2023 HTCondor European Workshop took place September 19–22 at IJCLab in Orsay, France, to join communities of high-throughput computing (HTC) users together. Attendees and HTCondor users have the opportunity to learn from developers and vice versa, HTCondor Core Developer Greg Thain says. During the workshops, “[I]nformation [is] going in all directions — developers to users, users to users, and users back up to developers,” Thain elaborates. Attendees discuss aspects of HTCondor Software Suite (HTCSS) and HTC that they like and areas that could undergo potential development.

This year, one featured talk was from the European Weather Cloud (EWC), part of the meteorological community, which just started using HTCondor, Thain mentions. In their presentation, Francesco Murdaca and Mike Grant discussed their challenges and current uses of HTC. Other HTCondor users like DESY and CERN also provided updates, challenges, and the scope of their current HTC uses.

Another highlight was this year’s “Lightning Talks” sessions, which gave individual attention to users as a way for them to highlight what works, what doesn’t, and what they’re trying to accomplish, HTCondor Technical Lead Todd Tannenbaum says. These lightning talks spurred spontaneous discussion. Also included in this year’s programming was a discussion of Pelican, a new system for sharing data, Thain reveals.

HTCSS provides distributed high-throughput computing (dHTC) resources to users in academic, government, and commercial organizations across the globe. High energy physics is a leading group of dHTC use, of which CERN in Geneva, Switzerland, is a major player. For high-energy physics, Thain explains that more computation needs to be done than can be accomplished in one physical area, so physics communities and member nations affiliated with CERN share resources with each other. However, HTCondor’s resources are not restricted to just these organizations — a broad range of scientific and research disciplines tap into its resources. “About 50% of the participants were regular participants — we’ve seen their faces at a lot of these workshops in Europe — but happily about 50% of the faces were new. So that was an encouraging sign, and we are making plans to have another one,” Tannenbaum says. “The audience has widened a bit from just the system administrators at these national labs that are doing the LHC computing to include a couple of commercial companies and other universities.”

The topics of discussion vary by year, Thain explains, depending on new developments or changes in the computing landscape, but are mainly driven by the Europeans. “One of the things we do in the year before is try and take the pulse of what’s new, what’s concerning, or what’s difficult and try to make sure that we have a workshop that addresses that,” Thain explains. “We’ve talked about the new tokens the last couple of years, and there’s been a lot of concern about electrical power, especially in terms of global events.” With the war in Ukraine and energy embargoes from Russia, electric prices have been less stable, Tannenbaum says, which is a big concern of European data centers. Security, energy management, and power-saving were big themes of this year’s workshops.

One of the popular workshops — and one that Tannenbaum looks forward to — is the “Show Us Your Toolbox” session. During this session, “…folks from all the different national labs [show] how they solve problems like monitoring their cluster, managing data, and interactive work. Just talking about what challenges they have at their site and their solutions to date inspires good discussion amongst the participants,” Tannenbaum shares. Other topics up for discussion included how German sites were deploying HTCSS, ways to improve upon the current HTCSS, and the money and resources users saved with HTCSS.

Another opportunity for users was participating in office hours, where they could take their computing issues to developers who work on HTCondor. For Tannenbaum, this is instructive because it helps him determine where people run into problems that he might not experience and understand which parts of HTCSS may need refining.

Planning for the 2024 HTCondor European Workshop is already underway, with the venue set for Nikhef, the Dutch National Institute for Subatomic Physics, in Amsterdam, Tannenbaum reveals. Feedback from the attendees of this year’s workshop provided insightful information planners will take into account when they meet in a few months to discuss next year’s workshop. “Looking at the feedback from the user surveys, we felt that this was the proof of the importance of in-person workshops,” Tannenbaum says. Restricting the workshops to Zoom or other online formats, like what occurred in 2020, causes the workshops to become less participatory and more reliant on passive listening. “It was much more a series of lectures. [The format was like] slide show, next slide show, next slide show, which is useful to people, but it doesn’t have that give and take and that everybody is sharing and thinking together,” Tannenbaum says of the online workshops.

Across the globe, similar workshops have popped up or are in the beginnings of development in places like India and Southeast Asia, which the European workshops have had a part in spearheading, Tannenbaum says. “[T]here’s a lot of opportunities to network with people and share good ideas. If people are in Europe, we’d love to see them next year in Amsterdam. It’s a great opportunity to have high-level conversations with other users. These last 10 years of meetings have come out of the work that was done almost 30 years ago,” Thain states.