Connect with RIPE 60 : Facebook Twitter RSS linkedin

These are unedited transcripts and may contain errors.

Notice: Use of undefined constant steno - assumed 'steno' in /var/www/html/ripe-60/steno-transcripts.php on line 24

The DNS session commenced on the 6th of May, 2010:

CHAIR: Morning. I am one of the co?chairs of the DNS Working Group. We have a packed programme and a packed hall and I see two or three places here and on this side.

Welcome, everybody, and this is the agenda, a little bit small but about the same as we had on the web. The scribe is Adrian and the ?? microphone etiquette is, this is recorded and streamed on the web, so please make yourself known and identify you on microphone, and to go over the last bit, to finalise agenda, depropped one thing out of Joe ably about the IDNs what is going on, which gives us slightly more room in the rest of the agenda. I see another five minutes we can skip. And matters arising from the previous RIPE 59, everybody has read it, of course, and are there any remarks about that or not? If not, we have another five minutes. And there were actually no items left off the last meeting, as far as I remember, but if we forget something, let us know and then...

So we didn't. OK. The first on the list is ?? going to report about what transpired at IETF with regards to DNS.

MATTHIJS MEKKING: So thank you, Jaap. So, what happened at the IETF, previous IETF regarding DNS, so the last IETF was in March, it was in beautiful place of Anaheim, got lovely pictures one here, this is Anaheim, but of course we are not there for the beautiful sights, we are there for meeting and two Working Groups there, it's DNS extensions to the DNS and DNS op, it's operational. First DNS E X T. Four topics there, for pour IB Ns, topic you might know is roll over and die, research that looks at resolvers that has invalid Trust Anchors configured and two documents that is update.

So, IDNs: There was a discussion going on, how to support idea internationalised domain names. DNS gets a different meaning for equivalence if you propose internationalised domain names. Currently, it's just case insensitive which is equal, for example the top level domain for Russia, if you write it like this, it doesn't matter. Russia has also got a proposed internationalised TLD P DB, stands for Russian Federation. You can up?put that in the DNS directly, this cyrillic script, but there is this is a puni code representation for that, which is this weird screen, X N ? P 1A I. Now, you have got two different zones actually, but you actually want to make them work the same.

So, how do do you that? There are actually two proposals already; one is clone the zone and twiddle a little bit with provisioning system and if you make changes in one reflected in the other or there is a redirection solution, proposed solution that kind of works kind of like D name, if you have a zone here redirect it to the original, in this case Russian top level domain. Most of the Working Group is still struggling with the problem, what do you mean with the same and what is a name, do you mean a host name or every name, does it work on a label level or can it be that a name of label length 1 is can point or act the same name of label length 3. So still uncertain what the problem is, how it should be solved and if this is a work for DNS E X T at the IETF.

Next topic is the roll?over and die, how to resolve this behave if you configure invalid Trust Anchor. This came from an incident that was, had resolvers configured still Trust Anchor still actually and it turns out that the resolvers current implementations are quite aggressive. If you misconfigure a Trust Anchor it causes the resolver puts trust in this so it doesn't see valid signature and it tries other name servers and tries a couple of times to see if that valid signature is out there so there is a large increase in the query load and the responses themselves is also large. What to do about that, resolvers need to be more conservative, I believe action has been taken in the software impreliminaryitations so they are more conservative, it raises questions should we have support more more widely deployed 5011, which is the automatically updating the Trust Anchors at the resolver or is there a need for history service so you can recover from still Trust Anchors.

There has been a little update in the DNSSEC base document, technical clarifications. There are some discussion about the CD bit, how you should handle that and perhaps some additional text is needed regarding the report I just mentioned.

DNS curve as presented there, it's a protocol that uses a he will incompetent particular curve cryptographics to improve confidentiality, integrity and availability. Working Group did not decide if this should be adopted, fairly I think because not many have read it yet.

So DNS O P was the next meeting at the IETF. A lot of topics there. Two presentations about research topics and a lot of status updates on documents.

First one was a research NSEC 3 hash performance, basically the question was: What is the worst thing that can happen if you increase the hash iterations, if you have a signed zone and using NSEC 3, you are in control of setting that number but also has an effect on the resolver so how big is that effect? The research looked at different kinds of number of iterations of course and also different number of key lengths, and observations were that key length has more impact on that resolver that hash iterations. Also you need a lot of number of hash iterations to have the performance at the resolver 150?plus. Well the authority daytive server has more to suffer if you increase the number of hash iterations. At a constant level of about 100 iterations it halves performance so that is good thing. The enten tiffs are well aligned, so that is nice.

There was a topic on IPv6 and recurveers resolvers. It turns out that if zone enables IPv6, it breaks a small percentage of the users, this percentage from statistics turns out to be like 470,000 users and that number looks rather large. How do we make the transition less painful, was the question there. They come up with this question, don't let users with broken IPv6 connectivity know about these records so they will never ask for it and never get it and try IPv6.

Five minutes left. I saw there are 20 minutes additional time, so...

From the community, they saw this solution and they say, well, this is ugly hack, but it might be necessary to make the transition less painful from what I have heard it bind 9 implement it had but it's very hard to turn on. Really, if you need this, you need to do a lot of work to your configurations and then you can use it. They want to stay with this, this is all about default. Also power DNS and secure 64 plan to implement this, so if you are in this situation you might want to look at these implementsitations.

Updates to the documents, there is update in the, a document that describes framework and can assist you if you are a writer of signing policies and practice statements regarding DNSSEC. A small update because dot S U reviewed this and incorporated changes and they requested that people read it because I believe there is a list of feedback going on here (little).

Another document in the DNS op Working Group, is trust history, a service to help servers recover from stale Trust Anchors. News on this topic is there are code 58, the name is TA link, Trust Anchor link and there is also like many documents and issue should we add the roll?over and die issue, should we make it more conservative.

What change in this document is the lifetime of keys is introduced, so that you can use the service if you have five or ten?year?old keys, you can set a lifetime and if it expires you have 3 options. No connectivity, no DNSSEC or you do out of band software update. Another change in the document is the document follows the 5011 automatic Trust Anchor,al rim, but it doesn't do the hold out time described in the RFC so you can follow regular roll?overs.

This document RFC 4641 is an update to the operational practices document and a lot of changes there are a lot of topics covered, recommendations for considerations whether you should use NSEC or NSEC 3 in your case, it's got a review of NIST, it's got additional text on switching DNS operators, how you should handle that and if they are not cooperative. And there is still a bunch of open issues on the website to the links on there. Two majors ones are key roll?over frequency, current draft says you should roll frequent so you have practice and you know how to do roll?over in the future, while nowadays, the thought is more don't do that frequent roll?over, only do it if it's necessary, if you need new cryptographic algorithm, for example. Who has to talk at audience, are we looking at large audience or mom and pop, as a quote from the Working Group zones? As you might consider or ?? wrong word ?? but, anyway, DNS operation on these zones are different so there are different best current practices.

Non?Working Group, a document, that is key timing describes all issues around key roll?overs, what time is necessary to keep keys in the zone, to keep keys use of signing, keep them post published, very technical document. Changes in that, there was a a request to describe all roll?over mechanism in detail, they did that. They decided to do algorithm roll?over in a different document because it's complex and requires a lot of text, and this is rather fair large document already. And the decision was made to keep it separate from the 4641 base document, that document actually got a reference to this one. The document is adopted by the Working Group, if I am correct; I don't recall.

Peter Koch: It is.

MATTHIJS MEKKING: Confirmtive. The last document in this Working Group is the IPv6 reverse DNS. It's a document that analyses possible ways for reverse DNS in IPv6. It does no recommendation; it just lists the possible ways, how you can do this. There were small changes but there was a discussion going on should we have the recommendation, what you should do and which ways you shouldn't work, and then there is the discussion if you, the recommendation is don't do this, what is the purpose of the document?

OK. I think we are fine, everybody still has Internet, but there was a question and answer section before all routes were signed during IETF and they had a small presentation about their, she showed that the UDP query rate is going down they had no real explanation for that but but they didn't consider it as a critical observation, and there was a small increase in TCP query rate, up to 80 queries per second, so doesn't look harmful.

Questions from the room were a lot, few of them I listed here. There was an issue on the improper maintenance of Trust Anchor, the roll?over and die issue, they said this needs more research and they are going to deploy 5011 to revoke keys. There was, there was the question of the frequency of the case K roll?over, the route is not going to be rolled quite often; they plan to do it five years, that is around the life time of the AS M the keys are stored on. There was a remark on missing statistics for IPv6 and several things, for example, the replies size test server showed there were some differences when the routes going to the DURZ and they suggested that the team, should have a look at that and if they can learn from that.

Here are the URLs you can go to to dive more into these topics.

CHAIR: Thank you Matthijs. There is a question from Deema.

AUDIENCE SPEAKER: Matthijs, just one remark: It's very incorrect example on dot RU and .FR because unfortunately this presentation will be published and people will think that we plan to close on it. It's different article.

MATTHIJS MEKKING: Sorry about that.

AUDIENCE SPEAKER: Please correct because it will abmisunderstanding.

MATTHIJS MEKKING: Maybe I can arrange to have a different example.


AUDIENCE SPEAKER: Just one remark, there is three proposals for the RIS zones and the last one is mine so ??

MATTHIJS MEKKING: Yes, there is the C?name, D?name proposal, it wasn't mentioned in the Working Group, I believe, and that is why I think I ??

AUDIENCE SPEAKER: I know, I just wanted to mention it because I wrote ID after Anaheim.

AUDIENCE SPEAKER: Peter Koch, co?chair of this Working Group and also co?chair of ?? a question to you Matthijs but a question to the you had aence, first of all we have been presenting this stuff for the last, I don't know, 25 RIPE meetings or so. We really, really would like to have feedback whether this kind of stuff is useful or you would like us to expand on this. There is a couple of documents there that have operator?relevant content, especially like the AAAA synthesis that was really, really controversies discussed at the IETF and also earlier this week and the draft that Matthijs mentioned by I guess it was Lee Howard on the structure of the reverse mapping for IPv6. So this is something that people in this room probably want to look at. Please give us feedback, whether we need to devote more time for that and then for the gymnastics because the air is draining here, how many of you do follow the IETF lists personally, like reading these lists?

CHAIR: A quick estimate about 40.

AUDIENCE SPEAKER: 23.7 percent or so, thank you.

CHAIR: Just before we ??

AUDIENCE SPEAKER: I wanted to add one other draft that was not mentioned at the presentation, that is a drafts on TCP TC I think is the name and it's actually, it's a mission completely out of the IETF, it's ex permal RFC proposing some experimental extensions to TCP to do some, I am mentioning it here because the stated objective of that draft was to reduce load on DNS servers as they have to perhaps go to more TCP as DNSSEC is being deployed, and this is just of course, just one idea or experimental spec, but and there is no IETF work around this at this moment, but if the DNS community feels that there is an issue with TCP load or the maintenance of TCP state and link the transactions and so on in DNS then of course we would like to know about that and perhaps start some activity. So if you have any data on, you know, increases in TCP transactions or things like that, then that would be useful.


CHAIR: Thank you Matthijs, and now we have an audit report about IETF, it's very specific about the registry operation the I i.e. if I remember correctly: For these people who didn't do it yet please download your slides and as Ed slides, his slides twice on the programme, used one for the session this morning because the first one doesn't work.

EDWARD LEWIS: I am not used to being in the first half of the agenda, I am used to being at the end. I have two topics to cover and these are mostly related to DNS in the sense that these are things affecting people who hold information that then goes into the DNS, typically registries but also registrars and LIRs and anyone else who keeps track of a database and so on.

So the first thing I will talk about is something called IR E which is an IETF activity. It was session at the last IETF, that means is that people came to the IETF significant we have a problem, some people care about it, we would like to potentially make this into a Working Group so it's in the first stages of being looked at. What this is data escrow and that is basically backing up the organisation. Not backing up what is in DNS or system backs?ups of hosts but what do you keep that goes into the DNS and if you have a registry database that would be the main item that would be escrowed. This is mostly important for if there is a failure of the organisation and someone else has to take over your job. If a TLD goes down, we have to trance near to somebody else, what is going to be taken over there is the copy of the database but in an escrowed form they are a could be ingested somewhere else.

The example I have here for registrar fails, who had that domain name? That could be who had that IP space and so on. It can go on to a couple of things. The IR E stands for Internet registration escrow. The acronym is R Y D E IRD E because they actually two names, they put/between it, it didn't really work well. The mail list is at the bottom. I have a list of mailing lists to go back to the. The status of this effort, the ?? one official BoF was held at IETF 77 at the same time. I would say it was inconclusive. The outcome was not to go ahead and make a Working Group but it also wasn't to not do this work. There are open questions whether this is appropriate work, is there something to solve and so on. Without going into that here because it's not so much DNS, I would suggest subscribe to the mailing list, see what is going on there, especially if you have an opinion on this topic, it's best to say it to the IETF channels, if you say something in the room here it's got to be carried there by somebody. It's much more effective to participate in the IETF directly through the e?mail lists than to just let it sit somewhere. We would like to know what is going on out there. Part of the IETF's job is to bring in in information but we really are structured to get this from the audience and not from liaisons and representations and so on. For Avenue 78, the next meeting, which is in July, I can't say what is going to happen there right now. I have no indication whether we have meeting there, BoF there, so on. It's really up to what the community says is is a good idea or not a good idea. Really want input for there.

The second half, I am talking about yet another mailing list so I have no pretty pictures of parking lots. Matthijs, you really stole the show there. Matthijs, pay attention. So at the corner of operations and DNS and so on, what we needed ?? for those who work in registry ?? I work for registry and DNS operator, we realise there is no really good way for that industry of registration to communicate, so we created a mailing list. We have lots of good sub mailing lists like this, Centre Attack, RIPE has good mailing lists for operations but there is no way for us to reach everybody who is involved in keeping track of who is who on the Internet, so we ?? new mailing list has come up. Unrestricted to membership, unrestricted to a region. We are trying to include anyone who is ?? who has any kind of input or, relevance to the registration activities. The list is the list; it's not an organisation, we are not having meetings. So there is no guaranteed outcomes. In fact most of the time we are going to point things off to where they need to go. So the first time ?? this is the mailing list, the name and the URL and this will appear two more times before I am done so I am going to keep going.

Why do I mention it here in this Working Group? Because DNS code writers are part of this. What happens in the DNS affects a lot of times the way we do registration, for example Matthijs was talking about IDNs, how do we make zones look the same. The code is in the right place, we can operate a certain way. It goes back and forth. Where do ?? one of the terms is cost shift all of this through the industry, who does the work to keep things the same. Now, code writers forks, LIRs would be really good to have LIRs because we would like to know issues involving reverse DNS operations. A lot of times we don't get good hands?on information to run the reversion DNS or if you want to run the reverse DNS the comments about Lee Howard's draft for v6. One of the for instances here which is the question I am most interested is in, how do we handle DNSSEC Trust Anchor, how get from the operated zone into the registry that they are part of. They are in a dotcom TLD, how do I get it over to Verisign or my registrar who goes to Verisign? It's a little complicated. On the other end if I am a DNS operator, how am I going to get this Trust Anchor out of some place? This is the interface hasn't worked out in the community, we have ideas of doing it but we kind of want to get this beyond the more known examples like I have just said.

Again that numbers. And I am going to say it again so I will keep going. More recent topics on the list. This was kind of where the list has started, the list has been in place for about a month. We just really had our first sense of discussion start in last day or so, actually. Some of the IETF effort about escrow and that, became a BoF already so it's kind of off the list on to the IETF area. We have had these topics listed here, ARIN has a interface talking about their interfaces to their registry. ID N best practices is is a topic that was mentioned once already, really is something is going to come across the board here. People wanton hear about events. Part of this is to help us not travel so much. Later I mention I get 6 could be copies of Joe ably's messages already, point people to where they can find out information and so on.

The bottom three are fairly large topics, those were the original ones, I ran out of space so they are just lumped together and by the way, one person was concerned when I said Whois; I mean anything to do with database inspection, Iris, if you like, Iris, and so on.

So this list, I am not trying to keep with other lists, this is trying to augment and unify the industry of registration to get everywhere. It's a low cost way for people to get their opinions to who do participate in other venues and so on. We also want too far place where we can say we have got an issue, my experience of the IETF goes back a long way and one of the problems I see is when we discuss protocols in the IETF we don't always make sure people who are going to be affected by it know about it early enough. I know people said the PPP when it came out ten years ago they didn't have the time or money to go participate in the IETF and I understand that. As a result, we kind of didn't have the best result possible; we have a good protocol but we could have been better. So I want this list to kind of be a way for people to lightweight get involved in what is going around in the other venues.

Reduce travel, go green, join this list. You don't have to travel if you are on this list. And one more time ?? so for the last time, these are the list name and the URL for the management interface to subscribe to REGOPS on top. The bottom one is escrow list, IETF list, IRE at, it has a web page to subscribe, through other mailing list methods that you are more familiar with, dash request and so on. So questions?

JIM REID: A bit concerned about the scope of this escrow exercise that is underway at the IETF. I notice you didn't say anything about financial data in the context of registry stuff. Is that going to be in scope or out of scope?

EDWARD LEWIS: I would say that that is probably not in scope but then again ?? since we don't have a charter, I can't tell you for certain yes or no. I would believe ?? probably against but to tell you the truth, it could come into play. If enough people want it to be there it could be brought in. Right now I think for the most part, people didn't want to touch that.


CHAIR: No more questions. Thanks, Ed.


Next ?? now, we have report from RIPE NCC about the well?known DNS monitoring system. And since that is live demonstration, you really need to do the swapping of some equipment here.

FRANZ SCHWARZINGER: Hi, welcome to my talk, I am Franz Schwarzinger from the RIPE NCC, I work in the information services department and the system architect there. And I am going to show awe little bit about the recent developments that we did with the DNSMON service.

First of all, a little bit of history: We started this in 2005 and the original purpose was kind of to figure out the, if clients can reach a certain DNS server. And TTM was used as simulated clients. And since 2005 a couple of advancements were added, we have we added IPv6 and ENUM support and recently we have been seeing some new challenges such as Anycast and DNSSEC.

So how do we actually measure those things? Well, it's quite basic, we have two types of measurements, we ask for the SO A records and any class of DNS servers especially the root zone, we ask for the ID server attribute as well. We do one measurement per minute, these are done with Poisson distribution so we don't measure exactly one minute but approximately every one minute, this is being done, the simplicity and Poission distribution is done so we don't interfere with the operation of the DNS server.

When we tried to reimplement DNSMON, make it a little bit better one of the first things we wanted to do is ask you guys who are using DNSMON who do you want from it, how do you like it. So last month we finally sent out the survey and asked people to give us their feedback and we mainly asked route server operators, our DNSMON subscribesers but also users who showed interest over time in DNSMON service and if you feel you haven't received the survey and would like to participate and fill it out, please come to me afterwards and we can arrange that. And what we ask people was basically three major questions: What is ?? what do you currently like about DNSMON? Why is it useful for you? Which types of graphs do you like especially, which types of graphs help to you debug your problems? The next point we asked was possible improvements and gave people a list of options what we think we could do for you guys, make them choose so that we can figure out which ?? which features we should put some priority but also ask you do you have any ideas. And we showed a new prototype interface and asked people to give some feedback on that. And what were the results:

I condensed this a little bit so we have four big points, one of the biggest ones was better performance, current DNSMON is is a little bit slow sometimes, this is because of the architecture but I will address that in a second, how we are trying to solve this; the next point was realtime data, people really want to know if a problem occurs instantly, how can I fix it; and the next two things are more on interface user, better drill?down approach so people can have a big overview of their domains than go into the servers and into the regions to see if it works in that region better than the other one; and people especially who are running Anycast, any traceroute functionality so you can in realtime click from that I want an traceroute to my server over there and see what pass to the server.

Let me start with the first two points ? dish forget about that, thank you very much for awful you who participated in the survey (all of). Your feedback is really valued.

I go to the first two points of this and like to address this a little bit, how we are trying to do this at the moment. These are Newark tech tour in the DNSMON. Up until now it's all been running on single boxes, we had two of them, to have a little bit distributed, but really all of user interface, the data insertion and gathering was all being run on one box. That is bit more fancy, this is the future, what we would like to do. And the upper part is basically the part where we focus on getting more realtime data, so we have the procession of the data done faster, and this works, we have of a control system here on the side, which is basically a message queue that gets, takes all the incoming new data from the text?boxes and then dispatches it into the processing nodes and the cool thing about these is that they are kind of built in a shared architecture, we can at any time add processing nodes subscribe to the message queue and if they don't have work to do they will take work from the messaging system. In theory we could even plug in my laptop and let it do DNSMON processing, this is quite nice.

Then, the process data is stored in round?robin data files in a big ?? some big disks and the lower part of this is the user side of things, so we really decoupled this from the processing so that we can have especially high performance on the user side so we have these plot generators which is one of the most processing intensive things in DNSMON and we can also scale them up to as much as necessary and we have them behind load balancer and then feed it into the user interface which you guys will see will be much faster than the current one.

So this was the system side of things. What about your actual user experience? Well, let's have a look at what our users currently don't like so much about DNSMON. One thing is they only have those three pre defined views so you can at the moment look at either a serve view, a domain view or a probe view but people want kind of a little bit more flexibility in that. Another complaint is actually very simple one, it's quite difficult to find your own server, we have this huge list of servers in DNSMON and it's quite difficult to find where your servers are. And the performance like I said is not always very good.

So I would like to show you one of our first steps at approaching a solution. And that will be in a little live demo. Let's hope it works. OK. This is how the new prototype interface looks. In the beginning we get an overview of all of ?? all DNS servers that we monitor. If you actually logged in as your user account then you will get an overview for just your domain so this is just a standard view that we have at the moment. We provide you with this really simple search box up here so finding your domain or your domain DNS server is simply a matter of typing something and Nick A T guys allowed me to show their domain. We will have a look at the A T domain. And we do a little bit of smart guessing here on what you will actually like to see. And please work. What is going on here? Well, I was clever you have no prepare some screen shots for you. I knew this would happen, but OK, let me show it to you in screen shots. So again, we start here. And then we get this nice little auto complete box where you can see all the service for A T or associated with A T and in the domain and you click "enter" and what should have happened is that you see this. So, a plot and all the dot A T servers are stacked up and it shows you for the ?? that for today. If you go further you can click here on last week, we have a few shortcuts for today, for the last hour, for the last week and last month, to give you a better overview of the time, of the time frame you want. And we also built something really cool in here, you can actually type a date or a time frame in here, so if you just type A T since February 2010, it will show you a plot since February 2010. You can also add say I only want to see it from this region just from one, so this search box is quite flexible and can be really fast navigating around this once you get used to it. If that is not enough for you, you have here a customised plot button where you can actually define the plot in much more detail. On the left side it's, those things are quite simple, you have the not time and the date range but on the right?hand side, that is where it gets interesting so we have this stack criteria, so what is this? Let's go back briefly. In this plot, we stack by service so we want to see one domain and awful the servers for that domain, they are stacked up each line. What you can do, you can switch the stack criteria and let's say you are not only managing one domain but a couple of domains, then you could actually change the filters down here and say show me not one but my three domains that I have ?? you would get a plot like that. And you see the A T domain, the dot come domain domain stacked in this plot. Yes, I guess you can imagine how flexible this is and how many ?? how much more cool things you can do with it. I don't have too much time to show you much more of this but you can visit me afterwards at the demo stand and I will show you more of what we have done in here.

That was the demo. Sorry for not being live but I think you got the idea, idea.

What are our next steps to actually deploy this cluster that we are building in order to allow for more scalability, hopefully then more probes, adding more and getting better coverage for Anycast service and another big step that we want to take, we actually want to improve the measurement methodology to allow support for new things such as TCP checks, DNSSEC or the RFC because at the moment we are only doing the ID dot server checks and SI D would be a nice thing to have for Anycast operators as well.

Like I already said, I will be manning a demo stand outside in the hallway just around the corner from here, I will be there all afternoon, you can come to me at any time. I wouldy especially like to invite our DNSMON subscribers to come see us in the afternoon coffee break so we maybe all get together at one time and have a chat together about how we can improve things especially for our subscribers. You can also discuss this on RIPE labs, yesterday we published a post going in a bit more detail about how the new interface user works and what kind of plans we have. If you want to get in touch with us directly, you can e?mail us at or me directly at That was it from me. Any questions?

AUDIENCE SPEAKER: Domain programme. I am interested in messaging programming system, the how is it implemented, the progressing leverage.

FRANZ SCHWARZINGER: We are use ago message system called Rabbit MQ, and then there is a framework on top that, but if you want to go into more detail with that, I'd suggest we talk in the hallway about it.

AUDIENCE SPEAKER: Rabbit MQ, I know is written ?? Rabbit MQ system is is a high performance written ??


AUDIENCE SPEAKER: The processes who are processing the Q itself are written ??

FRANZ SCHWARZINGER: Are no paste on ??

AUDIENCE SPEAKER: Interface between this and Rabbit MQ.

FRANZ SCHWARZINGER: Yes, standard interfaces that you can just use.

AUDIENCE SPEAKER: The framework who processes the messages is not written better ??

FRANZ SCHWARZINGER: No, the framework that ?? that uses the message system to dispatch those jobs is written in Poisson, the one we are using. I would be happy to show you that in more detail in the hallway.

AUDIENCE SPEAKER: Thank you very much.


AUDIENCE SPEAKER: New Zealand registry services. Have you considered providing an interface to access raw data?

FRANZ SCHWARZINGER: We don't have one at the moment, we are building this ?? the back end of this new interface is actually restful API that we could also provide to the public or to our subscribers. It is possible but we are not doing that at the moment. But the raw data is available for download on our website, on our FTP server.


CHAIR: You want to do more realtime. Will this mean that if I want to DDoS a name server, I need to see whether or not I have success on the website?

FRANZ SCHWARZINGER: That is the idea. I think as the plan is right now that we will provide the realtime data to our subscribers and I don't think our subscriber want to DDoS their own servers so I hope this will not be a problem.

CHAIR: Thank you. The interface looks much better than a couple of weeks ago. Any more questions? If not ?? thank you, Franz.


And the next in line is Anand and he is going to talk about reputation of the RIPE NCC name servers and other little details like route servers and stuff like that.

ANAND BUDDHDEV: Good morning, everyone. I am Anand Buddhdev and I manage the DNS services at the RIPE NCC, and I am going to tell awe little bit about what we have been busy with. First, just a quick introduction to the team, we are a small team at the RIPE NCC, myself and Wolfgang on the other side. You are all invited to feel free to approach us at any time to talk about DNS stuff. If you have any questions, queries, feel free to look us up.

The RIPE NCC DNS services department runs a variety of services. We operate K?root and we have 18 instances of this running in various locations around the world. We also run is he verse DNS services. We operate the parent zones of all the address space that we allocate from our region. We also operate secondary ccTLD service for those cc TLDs that are still developing or not technically able of running their own secondary services. We also operate the ENUM zone. This is ?? ENUM is a system that allows mapping of telephone numbers into resources. We also have an instance of an AS112 server at the AMS?IX, Amsterdam text, this is an instance that mops up all the queries for RFC 19, 18, PTR queries. We have been doing DNSSEC for a long time now, since 2005, we sign all our zones (Amsterdam Internet Exchange) and we have been publishing our Trust Anchors on our website for people to download and use and we provide internal services to the RIPE NCC, we manage all our forward zones like and stuff.

So, in the past few months, we have been very, very busy with K?root, obviously. We have been doing lots of things to prepare for the roll out of the DURZ, together with NL NetLabs we did a lot of testing of our K?root infrastructure and set?up to make sure that we were able to handle all the from a queries that were going to come out way, all the extra TCP stuff. And we made sure that we were ready for all this stuff by doing bandwidth upgrades so we could pump out more data, if necessary. We also upgraded our name server 3.2 .4 verse and this has some interesting options for TCP and E DNS buffer sites tuning.

In parallel with this we have also been busy with lots of data collection. We regularly collect all the P caps of all the queries to come to K?root, and at every DURZ roll out event we have been participating in a little mini style data collection whereby we upload our query traces for analysis. We have also been separately collecting priming queries and we have been uploading these, so we check to see if there were any interesting patterns, changes, as the DURZ was rolled out.

This P cap data has also been used by our science group to do some analysis and the results have been published on the RIPE labs website. And finally, we have been very busy also trying to reach out to the public to make people aware that, you know, the root zone is going to be signed and what that means for everyone. We have written some articles on the RIPE labs website for this purpose. We took the DNS reply size testing code and deployed that on K?root instances, as well, and we wrote or rather, Wolfgang wrote a very nice little Java appellate that people could download and check to see whether the resolvers were ready for DNSSEC or not. Here you can see a little screen shot of the replies tester. This is available from the RIPE labs website and when you run it, you can select a resolver to test. It goes away and perform some queries and then tells you what your resolvers is capable of and if you see any problems there, then you are encouraged to make sure that your resolvers are ready for DNSSEC.

So, K?root rolled out the DURZ on the 24th of March, that was during the IETF. We didn't see anything of significance in terms of query rates. Everything was stable, everything looked fine, but the one difference that we did see was an increase in TCP query rates. It's nowhere near as big as people might have thought. We went from a background rate of under five TCP queries per second up to about 40. I checked this morning and it was up at about 50 or so. So this has no negative impact on the performance of the DNS route server at all and has ?? yes, basically this the sky hasn't fallen down.


ANAND BUDDHDEV: Yet. We shall find out. We have also been busy with K?root expansion. Last year, in 2008 ?? sorry, in 2009 that should have been, we deployed an instance of K?root in Tanzania and we did this in cooperation with AfriNIC. We also intend to deploy a few more instances in the AfriNIC region. This is very interesting because, recently, there was a outage on the C come cable which is along the east coast of Africa and Tanzania was actually cut off from the Internet, but fortunately, because they had an instance of this route server, they were able to resolve local names and they came back to us and said that they were very glad they have an instance of this route server. And we'd like to do this in some of the other countries in Africa, as well.

AUDIENCE SPEAKER: Route servers for everyone.

ANAND BUDDHDEV: We are also talking to the people at LACNIC and looking into possible expansion into Latin America and people from the RIPE NCC region have also approached us to ask about more local instances in the RIPE NCC region.

We have also been very, very busy with DNSSEC, a little bit of history: The RIPE NCC has been doing DNSSEC since 2005, was probably one of the first deployments and that was all due to Olaf who was at the RIPE NCC then. So we have got all this old DNS signer infrastructure. It's basically a bump in the wire kind of model where we have our zones generated by a provisioning system and then signed by the signer and then pushed out on to the public facing DNS servers. These are just regular Linux servers. The tools we use are based on PERL and there is no secure storage for the keys, the skis are stored in the file system and they are not very secure and there is lots of manual processes in this entire infrastructure and it's prone to human error. We also roll out keys quite frequently, our KSKs every six months so that is quite frequent. So we want to improve all this.

Going forward, we have acquired DNSSEC signers from secure64. We are keeping the same model, the bump in the wire, so our provisioning system will not change; it will generate the zones as it does and these will be transferred into the signers, signed and then sent out to the public facing servers for publishing.

At the moment, we have a KSK roll over in progress and we are using the pre publishing method for this. The reason for this is that there is no simple way to import our old keys into the new signers, so what we have done is we generated new keys on our new signers, on 23rd of March, and we have published these Trust Anchors on our website, so that people can download them and prepare the resolvers. And on the 14th of June, we will retire the old keys and switch to signing with just the new keys. In parallel with this, we are busy with reviewing all our policies and procedures, we are generating a new DNSSEC practices statement and we are also thinking about the key lifetimes and whether we should roll as frequently as every six months.

We have also been busy with regular DNS infrastructure improvements. We have acquired a Shane knee new 32?bit ASN for this purpose. We are going to introduce multi?server architecture and this is for redundancy,; currently we have single servers which serve all our zones, so if there was a failure, then although the DNS is by design, redoesn't, there would be latency in DNS resolution so we want to take that away, as well. These new servers will serve the in? and IP zones. ICANN is currently busy with the process whereby these will be delegated to their regional Internet registries and eventually we will also move all the RIPE NCCs forward and reverse zones into this new infrastructure. Our first site is already operatal at the Amsterdam Internet Exchange and we are planning a second site later this year.

We have had some interesting experiences with this new 32?bit ASN and I am going to talk more about it in the closing plenary on Friday, so be sure to attend.

Just a little update on the ENUM zone. Since RIPE 59, there have been two new delegations, malaise I can't and Ukraine. There was also a request from Voxbone 8835100, this was approved but the delegation is pending. That is because the name servers were not yet answering so we haven't yet done the delegation. Four of the ENUM zones are signed, Poland, Czech Republic, Netherlands and Lithuania. We are also busy with changes to our provisioning system which translates domain objects in the RIPE database into DNS delegation, our current software is quite old and has no support for certain features, and we are replacing this legacy software with new stuff. Some of the features we wanton add is support for glue and DS records for early registration transfers. This is a proposal that is still actually being discussed among all the RIRs and when it is finalised, we will be making announcements to the DNS Working Group. We have a delegation checker which does pre?delegation checks when one tries to submit a domain object into the RIPE database. This also has some features missing, for example, it doesn't do very well if all the name servers in a delegation are IPv6?only, so we'd like to fix this. It also insists that if a DS record is present in the domain object then that key also be present in the zone, and this doesn't work for those users who would like to pre publish their DS records so we'd like to have support for things like these as well.

Finally, we have been running these DNS lameness checks for several months now. We presented our work at previous RIPE meetings and a lot of users came and said that this stuff was useful but they didn't like all the e?mail alerts that we were sending out; it was too in your face, it was quite spammy, so we went away and thought about this and we decided that it would be quite nice to integrate this DNS lameness data into our shiny new Netsense interface that our IS team has been working on. So here is a little screen shot, you can see where one has entered a prefix and there is some DNS issues that were discovered and these are presented at the bottom of the map. One can click on them to get more information about what the actual problem was. At the moment, we are querying over one million zone and name server pairs for the DNS lameness and we find that about 5 .5 percent of these are actually lame or don't respond. So that is not too bad, actually. And these lameness checks are continuous, we run them every month and publish some statistics on the RIPE website as well.

That was it. Any questions from anyone?

AUDIENCE SPEAKER: I have a question regarding the separate AS for the DNS infrastructure, what is the purpose and ?? what is the purpose and the benefit of having a separate AS for the RIPE NCC to run its DNS infrastructure?

ANAND BUDDHDEV: Yes, so the idea is that we can have independent routing policies. We don't ?? if we use the RIPE NCC AS, then we would have to apply the same peering policies as we have for that AS and that AS has a different purpose, so we decided we'd get a separate one, and also we want to Anycast with this, so administratively, it makes much more sense to keep it separate.

CHAIR: Two of my co?chairs at the microphone.

JIM REID: Anand, very interested to hear what is happening with reverse zone signing for ERX space and I realise it's complicated because it involves other interactions with RIRs specifically ARIN. Can you expand on information you gave. You say there is a project underway, you are discussing things with the other RIRs, have you a rough feel or a gut feel as to when that may be ready and if there is a possibility for members say of this organisation to participate in trials with a new system which will dealing with signing for reverse zones in ERX space?

ANAND BUDDHDEV: These discussions are happening at the RIR level and I don't have an exact time frame for when these discussions will be concluded, but I am hoping that by the next RIPE meeting, we may have something to tell you, but I think Andrei is coming up and he might have something to add to this.

JIM REID: If it must be like DNSSEC and it will be done within six months?

AUDIENCE SPEAKER: An dry, this is being discussed and that requires some changes in our provisioning system, from user experience point of view I don't think at least for the RIPE region users, it shouldn't have any changes. With regards to time?line, as I replied to the mailing list, maybe midyear we will have more accurate plans on the roll I couldn't tell of this service and how we tackle this address space. And that also goes in line that other areas are deploying DNSSEC, they are at different stages of the deployment and that also somehow determines the urgency of this activity.

AUDIENCE SPEAKER: You talk about earlier in your presentation about the multi?server infrastructure. I am interested in load balancing policy you are employing in this set?up. What is the software, what is the technique?

ANAND BUDDHDEV: We use O SB F based load balancing so you have a router that talks O SB F to all the servers and they all run Quaga and announce routes to the router. And if we want to withdraw any one box from this cluster with withdraw the O SB F routes.

AUDIENCE SPEAKER: Have you ever considered the solution presented called re/HRA*EUD D which is very strong and powerful tool in not only load balancing alert, your load balancing at layer 7 and is capable of load balancing almost every kind of protocol and traffic?

ANAND BUDDHDEV: We haven't looked at /KR*EUL A D and we are quite happy to look at it and I'd like to talk to awe little bit about that but I will add that O SB F load balancing especially for DNS works quite well because most DNS traffic is UDP?based. We already use this technique within K?root as well and it has worked very well over the last several years.

AUDIENCE SPEAKER: Thank you very much.

CHAIR: Any other questions? I have got a small clarifying questions, all this lameness checking that is only on reverse tree?

ANAND BUDDHDEV: Yes the lameness checks that we are running are only on the reverse tree.

CHAIR: Thank you.

ANAND BUDDHDEV: Thank you for listening.


CHAIR: Our next speaker will be a Peter Koch and he is going to talk about ex experience and experiments with DNSSEC in Germany.

Peter Koch: So good morning, everyone. Thanks for the introduction, my name is Peter Koch, I work fork DE?NIC, the test bed we have been running. There is a severe warning sign on this slide that you may see there that should warn you you encounter some of the marketing content, I am not talking about my name, but an engineer would never write write dot D E, be warned if you see these signs. Here we are. So this is for the marketing content. 250,000 domains secured by DNSSEC.


So now for the reality. This is not completely untrue, which is part of why you do marketing, right. Back to the route map. What we have been doing, the D E top level domain is, I guess we are back on the top list of the country code top level domains with roughly 13?and?a?half million domains and of course we haven't signed them all, so what did we do? Mid?last year, roughly a year ago, it was decided to come up with a test bed to get us some sense of what the customer, the environment, ISPs, the market, so to speak, and I warned you again, is going to take to support DNSSEC or maybe not. So, in addition to some planning and project management and before that, in December last year we started to deploy or we started actually before that, but we had deployed a separate infrastructure for DNSSEC test bed so that meant we have two locations and have some details later, giving access to a special version of the D E zone and in December we just published our unsigned D E zone to give people who wanted to participate a chance to trim their resolvers to take part in that, published in a separate infrastructure. Then in January, just after the beginning of the year, we started publishing a signed version of the D E zone in there on that dedicated infrastructure and yes, we did have these roughly 250,000 domains signed, even before we accepted DNSKEYs for registration, and I will show you later how that works. And then finally, in March, actually March 2nd, we started to accept DNSSEC key material for inclusion into the test bed and the DNSKEYs, the registration here and some of the domains or the zone maintainers decided to participate and we incorporated these into the signed test bed. The test bed is scheduled to run until the end of this year and now, even though Jim spoiled that, but this is one of the rare occasions, rare remaining occasions of bringing DNSSEC into the same sentence as six month, so we roughly six month and a couple of weeks to go.

So, some data points there. How did we set this up? We are currently in the process of rolling out a new version of our normal infrastructure, with kind of 16 locations serving 3 Anycast clouds, v4, v6 and everything like that, but of course we didn't deploy the DNSSEC test bed on that because that would have been too complicated and too expensive so we chose to have dedicated authoritative server clusters at fancy locations to coincide or co?locate I should say with our registration locations. That is Amsterdam and Frankfurt. And that has also has the property we have nice short RTTs for the participants that we would have expected to be mostly present in Germany and the surrounding areas. We also chose one remote location to have kind of, I shouldn't say bad connections, not blaming our providers there, but to give us some bandwidth delay product to play with to study the effects of having to transfer 1.3 gigabit zone file on the wire a couple of times a day, and we are seeing interesting stuff there.

We are publishing a signed version of the life D E zone so what you see in the test bed is just signed. What you see on the life infrastructure, just that we don't update that as often. We are using inSec 3 with opt out, and there is 3 products that we are aware that have would be able to participate in this test bed that is bind 9 ? 7 or 6 maybe because unfortunately that nice 56 pack was back ported to that old piece of software, it is unbound, people recommend 144, maybe I can come to that later and. Data changes, again for the marketing hat, this is updates, right. Happening twice per day on the real infrastructure we are doing that every two hours, that is 12 times a day so we are lag ago bit behind here and the main reason for that is not that the signing would take so long but the distributing the zone to that remote location can take a bit more, say, than two hours at the moment. The frequency of changes is going to increase during the rest of the test bed beyond the status quo and that also means beyond the status quo that we have in our real life environment at the moment. So we are approaching the real life environment from two sides here, from that particular project and there is some improvement coming out of the DNSSEC test bed project.

Signing details. Of course we have of a ZSK, KSK split. What the root does and probably everybody else does, software is based on a Java product, why that? Our software developers love Java, I love them, so that was the product of choice there. It is also that there was a nice product available or nice piece of software available written by David blacker from Verisign and we could make use of that and our software engineers added P K C is 11 support so we could use the hardware, also well?known products in this community, for its H S M features and you have all these fancy buzz words there. The signing is done or will be done, I should say, but again, weighing truth for marketing here a bit, we do have two registry locations and we plan to have a redundant set?up in both, that means two signing systems in Frankfurt and two in Amsterdam and then two to three cards we are struggling a bit there right now, per system, and currently we are running on a single card there that gives us a performance of roughly 1,600 signatures per second so we can sign a D E zone with 900,000 signatures in ten minutes or so. That is going to improve but since we are in test bed stage right now, we have ample time to play with this stuff.

The KSK, two K length, also not too uncommon. Signatures are generated in advance and we are using SC A 6,000. The DNSKEY apex RR set is only signed by the KSK and that is also common practice these days.

We are using NSEC 3 opt out, using salt, and 32 iterations and that is a bit more than everybody else did and that is exactly the reason why. We wanted to make a difference here, and not just follow blindly these parameters that everybody else used because should we have the necessity in the future to change these parameters, then we would like to be prepared and have this tested somewhere before anybody goes into production with these large numbers and for those interested I would like to refer you to the study on NSEC3 study. This stuff will be published in DNSSEC practice statement that is due in the first half of June, I'd say.

The Internet broke. OK. Now for the numbers. I said 250,000 domains. D E zone has a property that makes us special, of course everybody is special but we even more special there. In contrast to most of the TLDs, we are not only serving second level domains as referrals, we are not only registering delegations but for historical reasons we have these 250,000 domains in there that are authoritative in the TLD zone which means as soon as we sign that these domains are signed as well. And yes, I spoil the whole marketing, I am so sorry. This gives us, seriously, some interesting insights because we now have a big zone to deploy, a bunch of crypto noise so to speak, to distribute, which doesn't well compress which is interesting if you push this through VPNs and so we are going to have 400,000, 500,000 NSEC3 records due this authoritative data, a lot of empty non?terminals in there and you see the high watermark was end of April, we are having an interesting pattern in there, always at the end of the month some registrars seem to drop some of the domains maybe an artifact of the billing system and maybe colleagues in the billing department want to look at that so. We reached roughly 900,000 signatures in that zone at the end of April, and we are going back to that, you see the trend is growing.

How do you get DNSSEC key material into the test bed. Using TLDs work with registrars and so do we and that is what we do in the test bed. The DNSKEY and remember we are not working on the DNS record, we are choosing DNSKEY as the piece to register because that is what the client has in its hand without further ado. The DNSKEY will be subject to some technical protocol checks but then it is submitted into the production database. There is no special test bed database or anything; it's our life production database. That means we had to bump up our registration system, the whole registry interface is DNSSEC ready at that stage. We are using RR I, that is our special flavour and it tastes much better than EPP.

AUDIENCE SPEAKER: Like all German food.

Peter Koch: OK. Well you made me silent, first time in two years or so. The web interface that is also accession this realtime interface is DNSSEC ready so that registrars either those that do mass registration or manually can submit stuff into the test bed. The DNSKEYs will be immediately visible through the registry interface which is fine for the registrars but also in the information services so that when you have D E domain that you know is signed you can look it up through port 43 or through the web Whois, with the Webb Whois you have to click and accept on the terms and conditions and solve a capture and add some spices for the German food there. And then you get access to the domain and you can actually see the DNSKEY being in the registry. The only difference is we don't publish the records in the life DNS but only in the test bed.

So, we will skip that, you have seen DS records enough these days, I guess. Some prerequisites: We would like to see recommended that the SEP bit be set, we don't require that because we don't want to require people to have a split, it's just another check to avoid mistakes. We decided that the revoked bit must not be accept because nobody could explain why it could be sensible to register or generate DS record from rejoked key. We only accept DNSKEY with IANA assigned code points, that is RSA and DS A with different hash algorithms and may follow next as soon as we get our hands?on the library and Java libraries supporting this. Other key parameters must obey their particular specification, the key length has to be in certain ranges and we would like to see the SO A record validated against at least one of the submitted DNSKEYs, why at least, why not all? Well, we heard about DS record pre registration and that is basically the same reason here, it allows for the pre reg ration of not yet visible Trust Anchors and that might be interesting in case of hand over so can the old registrar have the ?? have their key submitted and that would allow for a smooth transition here.

So observations: Now, you saw 25 with a couple of zeros on one of my first slides. I have skipped those because they don't have been any meaning anyway, right? So the truth is we have 25 zones in their signed and participating. Well, reality it's quite double the number but the other 25 would account for our internal testing zones. One of the zones and I guess we are going to hear about that this afternoon, is from our federal government, so it's good that the government trusts us. And a couple of people that have been active on our mailing list and are probably are known in the community, we actually have one registrar that pub clickically admitted to supporting this stuff and we know some of the others. The test bed has not seen many queries at the moment and we would like to improve on that. We have seen roughly 600 different IP addresses querying but roughly below 10 per second and that is something in the order that you could answer on the phone. But other than that ?? I hope the ITU didn't hear that ?? other than that, no news is good news, but this is for the part could I actually publish on the slides; we have found some minor issues, not so much with our set?up but in the like interworking between various resolvers that is one reason I recommended unbound 144 over the previous versions. We had some strangeness there with our set?up that we fixed and unbound was reacting but that is all fixed, if you look at the unbound release notes, the test bed is mentioned so thanks IANA labs for that extra promotion here. So people can go there.

We on our website, we have configuration file snippets for all three products that we are aware of and of course, we also published the trust anger there so you can copy that and go participate with your resolver.

So back to the marketing now. 25 out of like 13 million, that is of course little participation. But then, again, a reality check: We would not expect to, everybody to participate in a test bed or everybody to be interested in DNSSEC in the first place, so it is interesting to look at how many domains, second level domains or zones I should say in the D E do have a DNSKEY at the apex and that is roughly 550, and it has changed over the last year maybe from roughly 400 to 450, so out of these ?? out of these 550 we have 25 and there is, well, not perfect but at least not as bad as 25 out of 30 million.

Next steps going to expand on logging and reporting and we hope there is anything to report there. We will increase distribution frequency and that is in cooperation with another internal project as I mentioned. The signing we are doing right now is an interim measure. We are going to engage and continue signing in a separate database and the database will spit out small increments, increment that we can feed into our infrastructure. We also published the test programme that will include an NSEC3 roll?over. We are most likely not going to engage in KSK roll over right now but we will definitely address operator change under DNSSEC. That is basically it. Please participate. If you have any questions, Jaap and Jim will prevent you from going to the mic. But find ?? wow that is a way of preventing everybody.

JIM REID: Thank you, Pete per. Two questions very quick ones: First one was you mentioned three products that you use, are they in the test bed just now or tried and rolled out and swapped out and varied between the three? You said you were ??

SPEAKER: We are not ?? we are providing the authoritative site. These three products on the validators' side that we know of if you have another recurver validator you are free to participate and we would like to learn to know about that. Unfortunately, we know it isn't there yet.

JIM REID: Next question is, the fact you say you are using opt out is that an indication that if and when D E gets signed for real production they are going to be using opt out or is that too early to say

SPEAKER: You may well think this but I could not possibly comment.

AUDIENCE SPEAKER: Ed Lewis: You may have already answered one of them, you mentioned that you are sending three gigabits to Hong Kong.

SPEAKER: 137 actually, that is the wire format size of the zone.

EDWARD LEWIS: Which is a lot, 1.3 or 3 ?? you mentioned signing the zone over and over again, and one thing we have been seeing is that there has been a lot of activity in signing zones by large zones, this whole transfer system, we took for granted and find it's actually a big bottleneck and so I would encourage, and you said you were going to sign incrementally, that is a good thing, start with that, reuse signatures, anyone who is writing their own code, reuse signatures, it's very important not to regenerate over and over again, if it's good for 30 days, use it for 20 days, but also, use incremental updates of things. Smaller changes don't send everything. We have seen people sometimes signing only part of the zone but signing everything anyway. So the zone transfer process, really don't take it for granted. And that is kind of ?? some of the comments you made made it sound like you may have to make sure that the zone transfer process is going to work well,y superb leer for the far distance ??

SPEAKER: Thanks for the clarification and I agree most all of what you said. The experience to accepted this stuff to a far away location is kind of a self induced pain to be prepared for the exceptional situation that we would have to transfer a zone into total, which happens every once in a while, right? Of course, the one basic goal of going to continue signing having these small increments and the continuous signing will of course introduce jitters that others have mentioned and will reuse signatures. This kind of set?up with the cheap way out or cheap way in here, because we knew we would have to improve that in the future. We will definitely not reply upon for every day production use. The way we do it today is using UNIX diffs compressed and I would encourage everybody to sign their zone if it's big enough and then try to compress it and go back to their textbooks and learn why compressing crypto material isn't really a good way forward. That is basically it, thanks Ed.

CHAIR: Thank you, Peter.


We managed to run out of time for people, but we have to do the announcements from ICANN ?? notice that there are already 3 in production since yesterday and still the sky didn't fall. I hope to see you all back after lunch and have a nice lunch and after that, a nice lunch Nap. Thank you.