3GPP HTTP model for applications module

A new traffic model for ns-3. The patch contains the following:

- Two new applications: `ThreeGppHttpClient` and `ThreeGppHttpServer`.

- `ThreeGppHttpVariables` for all sort of randomization in the model, e.g., packet sizes, off times, etc.

- `ThreeGppHttpTag` to be used on each transmitted packet to differentiate main and embedded objects and to allow the model to compute delays.

- A new test suite `three-gpp-http-client-server`.

- Typical helper class `ThreeGppHttpHelper`

- Some histogram figures to illustrate the distributions of the random number generators in the model. They are included in the Doxygen documentation.

This issue is an slightly upgraded version of Http model already announced about 1.5 years ago, but was not completed: https://codereview.appspot.com/233780043/.

[Tom Henderson, 2016-10-20 16:56:08]

I do not have any detailed comments right now; the code is clear, well-written,
conforming to ns-3 documentation standards, etc.

I think we should move to include this in src/applications.  Although I
understand the concern about it being dated (HTTP 1.1), as long as we are clear
about it, I think we would do better to include it since we don't presently have
any browsing model in the mainline.

Also, this model has been used by a number of standards development groups, so
it is probably an important reference model to have.

My main technical comment is that I would prefer to see what is in the HTTP tag
be implemented as an ns3::Header instead.  Headers survive serialization
boundaries, while tags do not.  The possible fact that the HTTP header does not
match the (simulated) payload size can be handled by including enough dummy
bytes and subtracting off the header size.  Example:  If I want to simulate a
100 byte packet, but want to include a 'real' IP header as part of this, I can
create an 80-byte packet with dummy data (e.g. Create<Packet> (80);) and then
add an IP header-- the Packet::GetSize() now returns 100 bytes.

My other main comment is that we should copy over some of the Doxygen
documentation to a .rst page for the model library, and we should provide a good
example, perhaps hooking the many trace sources to Gnuplot helper objects or to
the data collection framework to generate representative plots.

Finally, I was searching for a better name than 'ThreeGpp' model since that
doesn't seem to be very intuitive as to what it is.  I looked at the various
references to this and there doesn't appear to be a well-known name for it (it
seems to always be cited by reference to TR 25.892, so I came up empty).  I am
curious whether this is known by another name somewhere that is more descriptive
about the model itself?  Otherwise we can keep as is.

[jani.puttonen, 2016-11-04 19:05:37]

Hi Tom et al!

A new patchset (#2) provided for the 3GPP HTTP model for ns-3 just a while
ago. This version includes:

- Changed from using Tag to Header
- Added Sphinx documentation
- Added one example

I do not have currently any better suggestion for the name of the model. I
agree that ThreeGppHttp is not that good, but at least I'm not  aware of
any commonly agreed acronym for the model. Any comments we welcome!

https://codereview.appspot.com/314810043/


BR,
  Jani Puttonen

On Thu, Oct 20, 2016 at 7:56 PM, <tomh.org@gmail.com> wrote:

> I do not have any detailed comments right now; the code is clear,
> well-written, conforming to ns-3 documentation standards, etc.
>
> I think we should move to include this in src/applications.  Although I
> understand the concern about it being dated (HTTP 1.1), as long as we
> are clear about it, I think we would do better to include it since we
> don't presently have any browsing model in the mainline.
>
> Also, this model has been used by a number of standards development
> groups, so it is probably an important reference model to have.
>
> My main technical comment is that I would prefer to see what is in the
> HTTP tag be implemented as an ns3::Header instead.  Headers survive
> serialization boundaries, while tags do not.  The possible fact that the
> HTTP header does not match the (simulated) payload size can be handled
> by including enough dummy bytes and subtracting off the header size.
> Example:  If I want to simulate a 100 byte packet, but want to include a
> 'real' IP header as part of this, I can create an 80-byte packet with
> dummy data (e.g. Create<Packet> (80);) and then add an IP header-- the
> Packet::GetSize() now returns 100 bytes.
>
> My other main comment is that we should copy over some of the Doxygen
> documentation to a .rst page for the model library, and we should
> provide a good example, perhaps hooking the many trace sources to
> Gnuplot helper objects or to the data collection framework to generate
> representative plots.
>
> Finally, I was searching for a better name than 'ThreeGpp' model since
> that doesn't seem to be very intuitive as to what it is.  I looked at
> the various references to this and there doesn't appear to be a
> well-known name for it (it seems to always be cited by reference to TR
> 25.892, so I came up empty).  I am curious whether this is known by
> another name somewhere that is more descriptive about the model itself?
> Otherwise we can keep as is.
>
> https://codereview.appspot.com/314810043/
>



-- 
Jani Puttonen
Principal Scientist
Magister Solutions Ltd.
+358 44 5640814
jani.puttonen@magister.fi
http://www.magister.fi

[Tom Henderson, 2016-11-14 21:12:54]

I again just have some minor comments mainly to improve documentation and
example program.

Please note that on ns-3-dev, the test suite does not pass:

FAIL: Test Suite "Run #8 delay=3ms ber=5e-05 mtu=1460 IPv6" (0.040)
    Details:
      Message:   Unexpected number of web pages processed.

https://codereview.appspot.com/314810043/diff/20001/src/applications/doc/appl...
File src/applications/doc/applications.rst (right):

https://codereview.appspot.com/314810043/diff/20001/src/applications/doc/appl...
src/applications/doc/applications.rst:9: The model is a part of the applications
library.
Add here that it is based on a commonly used 3GPP model in standardization, and
provide a primary citation.

https://codereview.appspot.com/314810043/diff/20001/src/applications/doc/appl...
src/applications/doc/applications.rst:83: kept open until the client
disconnects.
Please describe in this section how the Mtu attribute works.  It is not really
clear what is going on with this attribute; how it is selected to be 1460 or
536, and the fact that the user cannot control this?

By the way, if TCP timestamps are used, 1460 byte application data units will
not fit into one IP packet-- any concern?

https://codereview.appspot.com/314810043/diff/20001/src/applications/doc/appl...
src/applications/doc/applications.rst:142: Many aspects of the traffic are
randomly determined by ``ThreeGppHttpVariables``.
Add "An separate instance of this object is used by the client and the server."

https://codereview.appspot.com/314810043/diff/20001/src/applications/doc/appl...
src/applications/doc/applications.rst:177: user to easily install
``ThreeGppHttpServer`` and ``ThreeGppHttpClient`` applications to nodes.
Add "The helper object can be used to configure attribute values for the client
and server objects, but not for the HttpVariables object.  Configuration of this
attr

Add also "The client and server provide a number of ns-3 trace sources such as
"Tx", "Rx", "RxDelay", and "StateTransition" on the server side, and a large
number on the client side ("ConnectionEstablished",
"ConnectionClosed","TxMainObjectRequest", "TxEmbeddedObjectRequest",
"RxMainObjectPacket", "RxMainObject", "RxEmbeddedObjectPacket",
"RxEmbeddedObject", "Rx", "RxDelay", "RxRtt", "StateTransition")."

https://codereview.appspot.com/314810043/diff/20001/src/applications/doc/appl...
src/applications/doc/applications.rst:202: $ ./test.py -s
three-gpp-http-client-server-test
please explain here what the test cases are doing:

e.g. a configuration is set up as follows .... there is a delay, ber, MTU value,
and IP variant (IPv4 or IPv6) configurable ... the test suite steps through
different combinations of these parameters for the purpose of ...


Also, if you have done any offline verification of the model against other
simulators or published results, please summarize what has been tested (i.e. how
does a reader know that this model is actually true to the 3GPP specification?"

https://codereview.appspot.com/314810043/diff/20001/src/applications/examples...
File src/applications/examples/three-gpp-http-example.cc (right):

https://codereview.appspot.com/314810043/diff/20001/src/applications/examples...
src/applications/examples/three-gpp-http-example.cc:71: Ptr<ThreeGppHttpServer>
httpServer = serverApps.Get (0)->GetObject<ThreeGppHttpServer> ();
presently, this variable is unused.  It might be nice to use this variable to
demonstrate a couple of things:

1) how a user can hook a trace sink to listen to a server event

2) how a user can access through the PointerValue the instance of the
HttpVariables class and make a change to a value there.

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
File src/applications/model/three-gpp-http-client.cc (right):

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
src/applications/model/three-gpp-http-client.cc:68: "Pointer to random number
generator.",
Fix documentation string.

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
File src/applications/model/three-gpp-http-client.h (right):

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
src/applications/model/three-gpp-http-client.h:366: /// The `Variables`
attribute. A random number generator.
Delete 'A random number generator.'

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
File src/applications/model/three-gpp-http-server.cc (right):

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
src/applications/model/three-gpp-http-server.cc:74: "Pointer to random number
generator.",
The documentation string is incorrect.

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
File src/applications/model/three-gpp-http-server.h (right):

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
src/applications/model/three-gpp-http-server.h:272: /// The `Variables`
attribute. A random number generator.
Delete 'A random number generator.'

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
File src/applications/model/three-gpp-http-variables.cc (right):

https://codereview.appspot.com/314810043/diff/20001/src/applications/model/th...
src/applications/model/three-gpp-http-variables.cc:45: m_mainObjectSizeMean =
10710;
Do not initialize attribute variables here, because they will be later
overwritten by attribute assignment.  It creates a false impression that the
user can change these values here and have them take effect, since they are
overwritten.

[Luciano Chaves, 2017-02-01 10:39:40]

Hi.

I was looking at this code and I got a question about the ThreeGppHttpHeader:
don't you think that would be better if the HttpHeader could be used as a real
header? It should be placed together with other headers in the simulator (inside
src/internet/model/), be compliant with some Http version (maybe Http 1.1), and
should be added to the packet by the HttpServer and removed by the HttpClient
(or vice versa). The main reason I'm saying this here is that using the proper
HttpHeader and setting fields inside it would allow wireshark to proper parse
the packet, and would make this model much more close to real world.

I found an very old code review that can help:
https://codereview.appspot.com/13486044/ This HttpHeader (I can't precise the
version) was used by another HttpTraffic generator that is available at
http://www.saulodamata.com/codes/85-http-traffic-generator-for-ns-3.html (I'm
have already used that traffic generator on some simulations and the traffic is
perfectly 

An option here would be adapting this code to use that header in the same way it
is done on the other traffic generator. I know that this would demand some code
refactoring, but I think that the  result will be better. I can help you and
give some more information on how that traffic generation works, if necessary. 

One last thing: about module name, I would go for something simple as
HttpWebBrowsing... ThreeGpp doen't reflect what is the module for.

Cheers,
Luciano.

[Tommaso Pecorella, 2017-03-02 00:32:12]

Hi Luciano,

I understand both your concerns, but I'm against them.

The "old" HTTP header is more realistic (true), but it has some huge drawbacks,
and they're not easy to fix:
1) it's text-based (like HTTP), and it requires a complex parser to have a
robust deserialization, and
2) it's variable-length (not a dramatic thing but adds complexity).

I'd be way more favourable to add an *HTTP2* header because it's still
variable-length but at least it's binary and well defined.

About the name, I'd keep this one *exactly* because it's not a generic web
browsing model: it's the 3GPP one (with all the pros and cons).

I'll try to check the code as soon as I can.

T.


On 2017/02/01 10:39:40, Luciano Chaves wrote:
> Hi.
> 
> I was looking at this code and I got a question about the ThreeGppHttpHeader:
> don't you think that would be better if the HttpHeader could be used as a real
> header? It should be placed together with other headers in the simulator
(inside
> src/internet/model/), be compliant with some Http version (maybe Http 1.1),
and
> should be added to the packet by the HttpServer and removed by the HttpClient
> (or vice versa). The main reason I'm saying this here is that using the proper
> HttpHeader and setting fields inside it would allow wireshark to proper parse
> the packet, and would make this model much more close to real world.
> 
> I found an very old code review that can help:
> https://codereview.appspot.com/13486044/ This HttpHeader (I can't precise the
> version) was used by another HttpTraffic generator that is available at
> http://www.saulodamata.com/codes/85-http-traffic-generator-for-ns-3.html (I'm
> have already used that traffic generator on some simulations and the traffic
is
> perfectly 
> 
> An option here would be adapting this code to use that header in the same way
it
> is done on the other traffic generator. I know that this would demand some
code
> refactoring, but I think that the  result will be better. I can help you and
> give some more information on how that traffic generation works, if necessary.

> 
> One last thing: about module name, I would go for something simple as
> HttpWebBrowsing... ThreeGpp doen't reflect what is the module for.
> 
> Cheers,
> Luciano.

[Luciano Chaves, 2017-03-08 14:59:51]

Hi

About the HTTP header version, I agree with you that HTTP2 would be better than
previous versions.
Anyway, despite the version to use, I still think that using a real header is a
better approach.
And about the name, you are right. 

Cheers,
--
Luciano Jerez Chaves
Assistant Professor at DCC/UFJF
PhD candidate at LRC/UNICAMP
http://www.lrc.ic.unicamp.br/~luciano <http://www.lrc.ic.unicamp.br/~luciano>
> On 1 Mar 2017, at 21:32, tommypec@gmail.com wrote:
> 
> Hi Luciano,
> 
> I understand both your concerns, but I'm against them.
> 
> The "old" HTTP header is more realistic (true), but it has some huge
> drawbacks, and they're not easy to fix:
> 1) it's text-based (like HTTP), and it requires a complex parser to have
> a robust deserialization, and
> 2) it's variable-length (not a dramatic thing but adds complexity).
> 
> I'd be way more favourable to add an *HTTP2* header because it's still
> variable-length but at least it's binary and well defined.
> 
> About the name, I'd keep this one *exactly* because it's not a generic
> web browsing model: it's the 3GPP one (with all the pros and cons).
> 
> I'll try to check the code as soon as I can.
> 
> T.
> 
> 
> On 2017/02/01 10:39:40, Luciano Chaves wrote:
>> Hi.
> 
>> I was looking at this code and I got a question about the
> ThreeGppHttpHeader:
>> don't you think that would be better if the HttpHeader could be used
> as a real
>> header? It should be placed together with other headers in the
> simulator (inside
>> src/internet/model/), be compliant with some Http version (maybe Http
> 1.1), and
>> should be added to the packet by the HttpServer and removed by the
> HttpClient
>> (or vice versa). The main reason I'm saying this here is that using
> the proper
>> HttpHeader and setting fields inside it would allow wireshark to
> proper parse
>> the packet, and would make this model much more close to real world.
> 
>> I found an very old code review that can help:
>> https://codereview.appspot.com/13486044/ This HttpHeader (I can't
> precise the
>> version) was used by another HttpTraffic generator that is available
> at
> 
> http://www.saulodamata.com/codes/85-http-traffic-generator-for-ns-3.html
> (I'm
>> have already used that traffic generator on some simulations and the
> traffic is
>> perfectly
> 
>> An option here would be adapting this code to use that header in the
> same way it
>> is done on the other traffic generator. I know that this would demand
> some code
>> refactoring, but I think that the  result will be better. I can help
> you and
>> give some more information on how that traffic generation works, if
> necessary.
> 
>> One last thing: about module name, I would go for something simple as
>> HttpWebBrowsing... ThreeGpp doen't reflect what is the module for.
> 
>> Cheers,
>> Luciano.
> 
> 
> 
> https://codereview.appspot.com/314810043/

[jani.puttonen, 2017-04-06 18:44:09]

Hi Luciano and Tommaso!

We agree that HTTP/2 would be better approach. However, I think it is somewhat
out of scope here, since we base our model quite directly on 3GPP specs. I agree
that the model may be a bit outdated, but still iI think it is quite widely
used. I would propose the proposed issues to be extensions implemented later on,
or even another HTTP model?

BR,
  Jani



On 2017/03/08 14:59:51, Luciano Chaves wrote:
> Hi
> 
> About the HTTP header version, I agree with you that HTTP2 would be better
than
> previous versions.
> Anyway, despite the version to use, I still think that using a real header is
a
> better approach.
> And about the name, you are right. 
> 
> Cheers,
> --
> Luciano Jerez Chaves
> Assistant Professor at DCC/UFJF
> PhD candidate at LRC/UNICAMP
> http://www.lrc.ic.unicamp.br/~luciano <http://www.lrc.ic.unicamp.br/~luciano>
> > On 1 Mar 2017, at 21:32, mailto:tommypec@gmail.com wrote:
> > 
> > Hi Luciano,
> > 
> > I understand both your concerns, but I'm against them.
> > 
> > The "old" HTTP header is more realistic (true), but it has some huge
> > drawbacks, and they're not easy to fix:
> > 1) it's text-based (like HTTP), and it requires a complex parser to have
> > a robust deserialization, and
> > 2) it's variable-length (not a dramatic thing but adds complexity).
> > 
> > I'd be way more favourable to add an *HTTP2* header because it's still
> > variable-length but at least it's binary and well defined.
> > 
> > About the name, I'd keep this one *exactly* because it's not a generic
> > web browsing model: it's the 3GPP one (with all the pros and cons).
> > 
> > I'll try to check the code as soon as I can.
> > 
> > T.
> > 
> > 
> > On 2017/02/01 10:39:40, Luciano Chaves wrote:
> >> Hi.
> > 
> >> I was looking at this code and I got a question about the
> > ThreeGppHttpHeader:
> >> don't you think that would be better if the HttpHeader could be used
> > as a real
> >> header? It should be placed together with other headers in the
> > simulator (inside
> >> src/internet/model/), be compliant with some Http version (maybe Http
> > 1.1), and
> >> should be added to the packet by the HttpServer and removed by the
> > HttpClient
> >> (or vice versa). The main reason I'm saying this here is that using
> > the proper
> >> HttpHeader and setting fields inside it would allow wireshark to
> > proper parse
> >> the packet, and would make this model much more close to real world.
> > 
> >> I found an very old code review that can help:
> >> https://codereview.appspot.com/13486044/ This HttpHeader (I can't
> > precise the
> >> version) was used by another HttpTraffic generator that is available
> > at
> > 
> > http://www.saulodamata.com/codes/85-http-traffic-generator-for-ns-3.html
> > (I'm
> >> have already used that traffic generator on some simulations and the
> > traffic is
> >> perfectly
> > 
> >> An option here would be adapting this code to use that header in the
> > same way it
> >> is done on the other traffic generator. I know that this would demand
> > some code
> >> refactoring, but I think that the  result will be better. I can help
> > you and
> >> give some more information on how that traffic generation works, if
> > necessary.
> > 
> >> One last thing: about module name, I would go for something simple as
> >> HttpWebBrowsing... ThreeGpp doen't reflect what is the module for.
> > 
> >> Cheers,
> >> Luciano.
> > 
> > 
> > 
> > https://codereview.appspot.com/314810043/
>

[Tommaso Pecorella, 2018-02-11 04:22:06]

Sorry for the epic time I took to check the code.

Most comments are trivial... but one.

I don't understand why the application should be interested in the TCP segment
she and the interface MTU.

If the problem is that the server doesn't know the client's request length, just
add the length in the header.
As a matter of fact, we shouldn't rely on TCP segment size because it could be
dynamic (sooner or later it will be) and about MTU... well, it's not constant
either (see IPv6's PMTU).

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
File src/applications/model/three-gpp-http-client.cc (right):

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
src/applications/model/three-gpp-http-client.cc:125:
"ns3::Application::PacketDelayAddressCallback")
This doesn't exist. You renamed it "DelayAddressCallback"

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
src/applications/model/three-gpp-http-client.cc:129:
"ns3::Application::PacketDelayAddressCallback")
This doesn't exist. You renamed it "DelayAddressCallback"

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
src/applications/model/three-gpp-http-client.cc:269: NS_FATAL_ERROR ("Client
failed to connect"
This shouldn't really abort the simulation. A failed connect could be a normal
event.
Please use a more graceful error type (maybe a NS_LOG_ERROR).

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
src/applications/model/three-gpp-http-client.cc:460: NS_ASSERT_MSG (packetSize
<= 536, // Hard-coded MTU size.
Why this limitation ?

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
src/applications/model/three-gpp-http-client.cc:508: NS_ASSERT_MSG (packetSize
<= 536, // Hard-coded MTU size.
Same comment as before.

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
File src/applications/model/three-gpp-http-server.cc (right):

https://codereview.appspot.com/314810043/diff/40001/src/applications/model/th...
src/applications/model/three-gpp-http-server.cc:222: UintegerValue (m_mtuSize));
Please NO.
You can't change a default value for any subsequent socket, because other apps
in the simulation could be heavily affected by this.

I know you're trying to re-set to its proper value afterwards, but even that is
a bug. Different sockets could have different mtu(s)
Instead, use m_initialSocket->SetAttribute (...)