# Community and Ecosystem for Large Scale Research Infrastructure ## Participants * Georg Carle * Philippe Buschmann * Johannes Zirngibl * Raphael Hetzel * Tim Betzer * Veronika Bauer * Michael Oberrauch * Hyerin Kim * Wei Geng * Mehmet Mert Bese * Jörg Ott * Srivatsav Chenna * Matthias Wählisch ## Key points - How to spend 50M on a shared Research Infrastructure - Different needs of the networked systems community to be supported - Different parts - Common stable part as core infrastructure - Part that can be reconfigured - Special equipment and special deployments - What kind of experiments to support - Performance experiments - Internet measurements (also distributed) - Local deployment of emulated distributed experiments - Innovative CDN - Investigate CDN behaviors - Set of emulated ASes as Digital Twin of significant parts of the Internet - What kind of data storage and sharing to support - What to care about when providing others access to own infrastructure - Project specific, requiring approval procedures - Legal questions, requiring legal expertise for user agreements and liability - Resource management - Reservations (Calendar) - Quota - Approval processes for large-scale resource requirements - Comparisons with other Research Infrastructures - Lessions to be learned from other research infrastructures - PlanetLab - Cloudlab, Chameleon, Colosseum - Specific properties of SLICES-RI and SLICES-DE - Blueprint concept - pos Plain Orchestration Service - Reproducibility and Artifact Evaluation Support - Local Experiments and Internet Measurements - Research Data storage and processing - Linking to data providers (ISPs, ...) and data processing - Education support: infrastructure and teaching material for courses - Community engagement to address different stakeholders - Community building - How to reach different stakeholders - PhD candidates / Scientists - Master and Bachelor Students - Corporations - Large Companies, SMEs - participating in collaboration projects - ISPs interested in Internet data and processing capabilities - Needs by Collaboration Projects - Community that can be reached by project results dissemination ## Notes - How to spend 50M on a shared Research Infrastructure? - Different Means - Planetlab Like Infrastructure - Storing and Sharing the Data - Artefact Evaluation / Reproducibility - Which Types of Measurements are we talking about? - Internet Measurements benefit from diverse locations - Local / Hardware measurements have specific requirements, but usually are location independent - Answer: Hardware should account for both, HW should enable local measurement, but could be used for Internet Measurements - Is it possible to emulate / create something like a Cloudflare Infrastructure? - Range of possible / wanted types of experiments is dicateted by the community - Static vs dynamic hardware pool? How to deal with changing hardware / with on- and offboarding? - Requirements - We will not get 50 Million € again => have to accomodate the needs of many groups that want to perform experimental research on networked systems - Need to tailor the HW to all of the needs - Industry: Restrictions on Sharing Data? - Need for an agreement - Data sharing - Ideas - IP - Is it still reproducible, if the HW is changed on specific project needs? - Example: Company wants to add their hardware to the rest of the nodes; they are added; later they do not want to participate again - Core Hardware Requirement vs. special attachments (IoT, ...) - Focus less on reproducability, but rather Artefact evaluation - Lower Bar, but much easier to achieve - Still a very good thing for community - Reproducibility - Only the same statistics? - Can mean multiple things - How does Artefact Evaluation improve by using the same HW? - How do we actually share the infrastructure? - Are there some minimal requirements for all sides? - Some common denominator to achieve compatibilty? - [RO-Crate](https://www.researchobject.org/ro-crate/) as an example - pos: Plain Orchestrating Service: https://dl.acm.org/doi/10.1145/3485983.3494841 - Is there a system to define a specific requirement and then allocate these resources? - Topology Map? - How can the cabling be done in a shared Infrastructure? If we need to have a specific cabling setup - Layer 1 Switching / automation - How much money do you want to spend on special purpose hardware? For many experiments Layer2 switching may be enough. - Specialised Hardware is planned in SLICES - also incorporate previous 5G/6G special hardware - What is the current concept for the core HW? - Different approaches: - High Performance Machines - GPU enabled - Connected with 100 Gbits switches - With some extend Layer 1 Switches - Larger Number of Cheaper Servers - Distributed Locations - Planetlab Style - Are Data Sharing Solutions planned? Or only Resource reservation for experiments? - Definitely also data sharing solutions! - How are other universities/research centers incentivized to join SLICES? - How can they access their HW? - What are their time sharing quotas? - Similiar to RIPE Atlas Quotas? (Accumulate when giving your infra, spend when using others) - Is the credit only generated when People use your cluster? - Efficient Resource use from funding money: Compared to three year projects. - Who will be users / community? - What about fairness? - Small chair providing hardware -> more credits to be able to access hardware? - Quota for excellence vs. quota for sharing - How is the access controlled handled - Access control to Infra - Access control to results - Example: MPI / Siemens has strong export regulations - Would limit cooperations very strongly - Internet Measurements: Limits <-> Opportunities - Laws - Regulatory Stuff regarding Internet Measurements - Large Grey Area - Need a system to vet projects executions - Liability needs to be transfered via the legal system - Grey area in the law: see VPN operators, TOR end nodes - Idea: - Make the system more lean and a bit more removed from the infrastructures that already exist in these locations, have dedicated IP ranges - Maybe from TUM? - What is the unique benefit that comes from the actual HW instead of just using the money in AWS credits? - Special HW - Vantage Points - Less Work - Research collaboration - Community in agreement - Open source / openness - Using AWS as an alternative? - US company - Other complexities - Is there a wish to use a US Company in the current time - Include the topology of a network setup inside the platform so that people know how to use the network - [YANG](https://developer.cisco.com/docs/nso/guides/the-yang-data-modeling-language/) Model - Features need to incentivized so that in the short term people execute it - Long term thinking? How to do in academia? <!-- ## Goals? --> ## Second Day Discussion - Collect problems worth discussion: - Who is responsible when Internet Measurments are executed on hardware on shared Infrastructure? - Can we solve this question? - What is the infrastructure really about? Is it Planetlab 2.0? Is it more? - Answer: It is more, but have it as an aspect - Retrospective analysis of what went well and bad from Planetlab? Learning from the past. Published report of lessons learned unclear - Get insights from the people that operated planetlab - User perspective exists - CloudLab from the US may have a perspective, because it is still running - Chaemeleon also exists - Several others also exist - FI-Lab operators - One bad experience from Cloudlab - Interconnect and side-effects from switches are not well documented - Are the different sites connected? - A: for certain sites there are dedicated inter connections - TUM <-> TUD 100Gbits (6G Live, German Telekom) - Opportunistically use inter connections that already exists - Is there an use case for that? - Can we emulate a Hyper Scale Infrastructure? - Establish a list of scale large scale experiments to show what is possible? - What is the goal of the Infrastructure? - Digital Twin of the Internet? - Emulate a large part of the BGP network to see what is possible, what breaks if you change something - Use something like the Mini Internet - Virtualize the Digital Twin -> Why have it as physical HW? - Focus on a subsets of routers - 400 Student in groups course needs a large amout of HW/memory - ETH Zurich Mini-Internet project (Laurent Vanbever, open source FRR routers in Docker container) https://2024-mini-internet-instructions-netd-tud-teachin-8c372f604af42d.gp.hrz.tu-chemnitz.de/ https://github.com/nsg-ethz/mini_internet_project/tree/master - another not related project: https://mininet.org/ - Mini Internet people where surprised that it runs at this scale - Needs multiple hours to setup/boot - Peering Testbed as an example - Limits? - Very different infrastructure - Single Purpose: Investigate BGP Behaviour - Runs in the Internet - What is the latency between the devices? - nano seconds - There is a need for low latency between devices - Input: Only build the infrastructure that is actually requested - The setups are very different depending on the use cases - Internet measurements - Internet simulation - Low latency network - Compute - The testbeds are connected by default by an overlay - What are the features/requirements that pos (The testbad management software) needs to fullfill - What is the difference between pos and a open stack VM application - Requirement: No limitations for bare metal machines - Reproducability - Questions about POS - What can pos do? - What are the defaults that POS sets? - How are experiments done with POS - Linux Images are preserved with pos, and can just be reused - Input: Building a community needs to start right away - PHD Sommer school - Outreach to operator community/target groups - Involve Industry to get long term support - RIPE meetings for example - Input: Show the impact of the infrastructure and demonstrate what was possible with the testbed, which was not possbile before. Impact on Society, not papers in a conference. - Community Building Ideas - Teaching Instructions for second year BA students - If the infrastructure is always there, the hurdle to overcome may be lower - Students groups may be have hardware needs that we may fullfill - BA/MA thesises - POS seems currently limited to TUM Net Chair - Dedicated Company Clusters as a compatible starting point - Convince companies to join the initative, by convincing the companies that the research infrastructure is worth it - Start Up from TUM may be offered some of the HW? - Input: Do not just give them access to hardware to decrease the time of first experiment, but also make it possible to test out pos itself - Open Source Licensing can also be restrictive - MIT is often better/easier (for companies), GPL is more restrictive/can cause issues - Going around the table: - Denso - Happy with compute - Strict about IP - Publish Code after paper publication/patent acception - Internal progress - if there is a common project, it should/may be feasible to share infra - Siemens - similiar process: invention disclosure - Time depends on the need - [Fiware Foundation](https://www.fiware.org/foundation/) + [Fiware Foundation Success Story](https://digital-strategy.ec.europa.eu/en/news/fiware-european-success-story) - Was there a discussion with industry or is the focus more academic? - Talks with few industry people - Input: EU stuff often demand community engangement - those need to be nice to use to find adaption, otherwise people give up - Solve the demands of EU reviewer easier - Offer the community to solve that demand - Enable discussions, easy integrations, other community aspects - Instead of integrating in a complex system, make it possible to use in pos, which then fullfills the community demand - EU Project: Edgeless - [aerOS Project](http://aeros-project.eu/): Contributed to the Eclipse Foundation - Marketing is also important - Input: Some do not want to publish, becaue it means to maintain the software and there are no recources for that, so they cannot publish because it would look bad if it does not work - Artifact Evaluation: Badge: Is executable on that testbed - Define the state of the software and state that it can be executed on the infrastructure - Stakeholders: - PHD Students - Industry - Large Partners - Small / Medium Enterprises (SMEs) - Start Ups - Research Infrastructure - Artefakt Evaluation (independend of PHD Students) - Managing of Research Data - BA/MA Students - Getting them early - Operators? - Benefit from research outcomes from the infrastructure - Verticals? - Somebody, who want to be a part of the network - Attach their stuff - Provide an interface - Different flavor of AWS/Hetzner - Example: [Edgar](https://www.mos.ed.tum.de/ftm/forschungsfelder/team-av-perception/edgar/) Car - Define a model to sustainably run the infrastructure - Get external money - Demonstrate the benefits - Other research areas outside of networking - Robots people - ROS / ROS2 - Connect the robot to the testbed? - Safety concerns, not just a server, but rather moveable machines - Blockchain community - TEEs in the cloud - Getting HWs TEEs - Example TUM Blockchain Club - They offer (student) competition - May not have that much money to buy AWS Credits - There are test networks that offer the ability to test blockchain things (With quotas) - Machine Learning People? - Are always happy with more GPUs - But may not be suitable - May drive some KPIs up - Input: Computational Power Cost may also be a problem for cooperations - Universities do not (yet) pay for the power - Companies do * Unique Point: Combine different testbed ideas/patterns * Interconnection of Sites with dedicated Bandwidth * vs. emulating WAN interconnections at a single site to perform locally an emulated distributed experiment * Experiment of being like a cloud provider ##### # References 1. [MIR3 Brainstormed Topics](https://hedgedoc.cit.tum.de/8GbMKDg2Sk6eZpLmXtkn7g?view)