Code Brain

Recently I needed to write some code to query a list of WhoIs servers to find the registration date of each url in another list. I needed to write it in VB.NET so that it would link in with our VB ASP.NET code.

My first approach was to grab a list of whois servers from a flat file and just do a for loop to go through them all until I found the information I was looking for. The problem with this is that it is horridly slow and can take a very, very long time. It was suggested to me to process them in parallel so I did some searching through the web to find a good way to do just that and what I found was not only amazingly simple but also very, very fast. Microsoft has a Parallel.For and Parallel.ForEach. I also needed to know stop the loop when I get the information I need, which is where MSDN’s article on how to stop or break from a Parallel.For loop came in handy.

Notice how Stop and Break are two different things here. The basic difference is the time at which it stops threads running the loop that aren’t the one that the stop or break is being called by.

Stop tells the other threads, including ones that may be executing items after the one that is in the current thread, to finish their processing and stop when convenient. So let’s say that we’re doing a for loop from 0 to 9. 10 items. We might have something similar to the following:

Parallel.For(0, 10, Sub(i, loopState)
                               'processing code here
                               ' Somewhere in here we call:
                               loopState.Stop()
                    End Sub)

Basically, you’re passing an anonymous subroutine to the Parallel.For() function. So, passing a function to a function. That function takes two paramaters: the index (in this example i) and a ParallelLoopState object (i used loopState). The ParallelLoopState object is what you use to call the Stop() method. So for example’s sake, lets say we currently have threads with the index 4, 5, 6 running, and loopState.Stop() gets called from thread 5. It wont start any new threads with any new indexes, but it will wait for both for and 6 to finish what they were doing before finally stopping.

However Break tells any threads that are executing past the current index to drop whatever their doing, but lets anything previous to the current thread to finish before completely ending the loop. So going with the same example, running threads 4, 5, and 6, thread 5 calls loopState.Break(). Doing this lets thread 4 finish, but kills thread 6 immediately.

So now that we understand how the basics of these loops work, lets move on to the whois query part.

I basically wrote two functions; one to query all the whois servers for one particular domain in parallel, and one to execute the previous function on a list of domains I need the information from in parallel. The first function looks something like this:

Public Function ParallelQuery(ByVal domain As String) As String()

        Dim sr As StreamReader = New StreamReader("serverlist.txt")
        ' Read in the entire server list
        Dim whoisstring As String = sr.ReadToEnd()
        ' Split it by new line
        Dim whoislist As String() = whoisstring.Split(New Char() {vbCrLf})

        Dim returnData As String = ""

        Dim endData As String() = Nothing

        Parallel.ForEach(whoislist, Sub(d As String, loopstate As ParallelLoopState)
                                        Try
                                            returnData = ""
                                            d = d.Trim()
                                            ' Create a new TcpClient
                                            Dim tcpClient As TcpClient = New TcpClient(d, 43)

                                            Dim networkStream As NetworkStream = tcpClient.GetStream()
                                            If networkStream.CanRead And networkStream.CanWrite Then
                                                Dim sendBytes As Byte() = Encoding.ASCII.GetBytes(domain + vbCrLf)
                                                networkStream.Write(sendBytes, 0, sendBytes.Length)
                                                Dim bytes(tcpClient.ReceiveBufferSize) As Byte
                                                Dim recvSize As Int32
                                                recvSize = networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))

                                                While recvSize <> 0
                                                    returnData += Encoding.ASCII.GetString(bytes, 0, recvSize)
                                                    recvSize = networkStream.Read(bytes, 0, CInt(tcpClient.ReceiveBufferSize))
                                                End While

                                                If returnData <> "" Then
                                                    ' This is where you do your processing of the return data from the request
                                                    ' I am using a regex expression to get the creation date.
                                                    Dim regex As New RegularExpressions.Regex("creation.date:\s+(.+)", System.Text.RegularExpressions.RegexOptions.IgnoreCase)
                                                    Dim match As RegularExpressions.Match = regex.Match(returnData)

                                                    If match.Value <> "" Then
                                                        tcpClient.Close()
                                                        endData = {domain, match.Groups(1).Value}
                                                        ' If it finds what I need I can stop the loop.
                                                        loopstate.Stop()
                                                    End If
                                                End If

                                            End If
                                            tcpClient.Close()
                                        Catch ex As Exception
                                        End Try
                                    End Sub)
        Return endData
    End Function

The function that ran those queries in parallel looks like this:

Public Function processList(ByVal list As List(Of String)) As DataTable

        Parallel.ForEach(list, Sub(e)
                                   ' This returns the domain and creation date as a string for each one
                                   Dim res As String() = ParallelQuery(e)
                                   ' I can now do whatever I need to with the creation date

                               End Sub)

        Return dt

    End Function

You’ll notice that I used Parallel.ForEach loops in these functions. I didn’t need to use an index and instead thought it would be easier to have the object I’d be using inside the function passed to the function in place of the index. It’s relatively the same thing as the Parallel.For loop but you can use objects as the first parameter to the anonymous subroutine instead of just looking for an integer

So there you have it. I love .NET for creating this. AND it automatically chooses how many threads to use based on your system resources available. These methods make parallelism so easy it almost feels like cheating.