Netdevice queue feedback support

Dear all, here is the draft of the netdevice queue support. For some context information, see http://groups.google.com/group/ns-3-reviews/browse_thread/thread/6359c7c94a334f03/

The current patchset contains (a) the API addition to netdevice and (b) an implementation for Wifi scenarios. The patches 003-* and 004-* at http://code.nsnam.org/rmerz/ns-3-netdev-mq/ contain the test code.

In particular, I would need feedback/help on the QueueState structure definition (i.e., am I doing it the right way).

My objective is to have this feature merged in ns-3.12.

Thanks for your help and time.
Ruben

[Tom Henderson, 2011-07-26 14:30:00]

Is there a plan for allowing qdiscs someday in the NetDevice?  We have
previously documented that NetDevice::Send() is analogous to dev_queue_xmit()
but all of these new methods introduced here are really more akin to
hard_start_xmit() modeling.

Should we provide a separate (Hard)StartXmit() method and document that this new
queue flow control API pertains to that method, and not necessarily to
NetDevice::Send() (which may support qdiscs in the future).  We could document
that NetDevice::Send() is analogous to dev_queue_xmit() (i.e. it does not
receive fully formatted frames) but NetDevice::StartXmit() is analogous to
hard_start_xmit() and accepts frames ready for wire transmission?

http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
File src/network/model/net-device.h (right):

http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
src/network/model/net-device.h:339: virtual QueueState GetQueueState (uint32_t
index) = 0;
I think it would help to explain a bit in the doxygen what this index is.  I
think the answer is that there may be multiple queues, indexed in a way that is
specific to the net device, and even though the NetDevice API doesn't allow the
user to specify which queue the packet will get filtered into, if the upper
layer somehow knows this information, it can discern what the possible fate of a
packet would be, if it were to be sent into the NetDevice.  Is this correct?

http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
src/network/model/net-device.h:347: */
Why is this needed?  Can't GetNQueues() and GetQueueState() provide the same
information?

http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
src/network/model/net-device.h:353: * queue index.
Why not return the QueueState also in the callback, to avoid the client needing
to call back into the device with GetQueueState()?

[Mathieu Lacage, 2011-09-14 06:47:43]

I am sorry if this has been discussed before but I reviewed this carefully based
on the latest kernel headers. The biggest differences I see here are:
   1) the Linux kernel allows the driver to decide which queue to use to output
a specific packet _but_ it is allowed to communicate this information to the
higher level layers. i.e., PickTxQueue would take a Ptr<Packet> and return a
queue index.
   2) the linux kernel allows the higher level layers to pick which queue to use
for a specific packet. They store this in one of the skb fields.

So, what is the impact from the perspective of integrating a linux kernel ip
stack on top of this ? I think that it is pretty much nil because the only user
of this API in the linux kernel is the 802.11 stack (the packet scheduling API
has a similarly-named function select_queue but it uses to manage the scheduler
queues, not the device queues).

So, I think that the above is good. It would be really nice if frederic could
try it out to make sure it is ok because this has a fairly large API impact.

http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
File src/network/model/net-device.h (right):

http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
src/network/model/net-device.h:347: */
On 2011/07/26 14:30:00, Tom Henderson wrote:
> Why is this needed?  Can't GetNQueues() and GetQueueState() provide the same
> information?  

+1

http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
src/network/model/net-device.h:353: * queue index.
On 2011/07/26 14:30:00, Tom Henderson wrote:
> Why not return the QueueState also in the callback, to avoid the client
needing
> to call back into the device with GetQueueState()?

In general, I think that it is better to make such callbacks just query the
object that generated the event for anything they need. i.e., in this case, what
you need is to provide at least one extra argument that is the Ptr<NetDevice> on
which the event occured.

[Frederic U, 2011-09-14 07:49:33]

On 2011/09/14 06:47:43, Mathieu Lacage wrote:
> I am sorry if this has been discussed before but I reviewed this carefully
based
> on the latest kernel headers. The biggest differences I see here are:
>    1) the Linux kernel allows the driver to decide which queue to use to
output
> a specific packet _but_ it is allowed to communicate this information to the
> higher level layers. i.e., PickTxQueue would take a Ptr<Packet> and return a
> queue index.
>    2) the linux kernel allows the higher level layers to pick which queue to
use
> for a specific packet. They store this in one of the skb fields.
> 
> So, what is the impact from the perspective of integrating a linux kernel ip
> stack on top of this ? I think that it is pretty much nil because the only
user
> of this API in the linux kernel is the 802.11 stack (the packet scheduling API
> has a similarly-named function select_queue but it uses to manage the
scheduler
> queues, not the device queues).
> 
> So, I think that the above is good. It would be really nice if frederic could
> try it out to make sure it is ok because this has a fairly large API impact.
> 
> http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
> File src/network/model/net-device.h (right):
> 
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> src/network/model/net-device.h:347: */
> On 2011/07/26 14:30:00, Tom Henderson wrote:
> > Why is this needed?  Can't GetNQueues() and GetQueueState() provide the same
> > information?  
> 
> +1
> 
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> src/network/model/net-device.h:353: * queue index.
> On 2011/07/26 14:30:00, Tom Henderson wrote:
> > Why not return the QueueState also in the callback, to avoid the client
> needing
> > to call back into the device with GetQueueState()?
> 
> In general, I think that it is better to make such callbacks just query the
> object that generated the event for anything they need. i.e., in this case,
what
> you need is to provide at least one extra argument that is the Ptr<NetDevice>
on
> which the event occured.

Yesterday I spend much time to understand this review, and finally I have more
questions than responses :

  1. the concerned queue are the equivalent of queue disciplines found in the
Linux Kernel ?

  2. if YES will NS3 manage this queues under netdevice or elsewhere ?

  3. concerning DCE kernel usage I see that today we call the
NS3::NetDevice::Send after the processings of queues so
     from our point of view the NS3::NetDevice::Send is the equivalent of the
Kernel  ndo_start_xmit which is low level
     and do not take care of queues, then it should also return a code like :
NETDEV_TX_OK or NETDEV_TX_BUSY.

  4. So if the point 3 is true in DCE / Kernel mode the qdisc will be still
managed in linux kernel and not in NS3 ?

Else I would like to have the Ptr<NetDevice> parameter in the Callback setted by
SetQueueStateChangeCallback because recently I use AddLinkChangeCallback using
an annoying callback without parameters.

Meanwhile I will test the current patch proposition, and I think it will be OK.

[Frederic U, 2011-09-16 14:56:42]

On 2011/09/14 07:49:33, Frederic U wrote:
> On 2011/09/14 06:47:43, Mathieu Lacage wrote:
> > I am sorry if this has been discussed before but I reviewed this carefully
> based
> > on the latest kernel headers. The biggest differences I see here are:
> >    1) the Linux kernel allows the driver to decide which queue to use to
> output
> > a specific packet _but_ it is allowed to communicate this information to the
> > higher level layers. i.e., PickTxQueue would take a Ptr<Packet> and return a
> > queue index.
> >    2) the linux kernel allows the higher level layers to pick which queue to
> use
> > for a specific packet. They store this in one of the skb fields.
> > 
> > So, what is the impact from the perspective of integrating a linux kernel ip
> > stack on top of this ? I think that it is pretty much nil because the only
> user
> > of this API in the linux kernel is the 802.11 stack (the packet scheduling
API
> > has a similarly-named function select_queue but it uses to manage the
> scheduler
> > queues, not the device queues).
> > 
> > So, I think that the above is good. It would be really nice if frederic
could
> > try it out to make sure it is ok because this has a fairly large API impact.
> > 
> >
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
> > File src/network/model/net-device.h (right):
> > 
> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > src/network/model/net-device.h:347: */
> > On 2011/07/26 14:30:00, Tom Henderson wrote:
> > > Why is this needed?  Can't GetNQueues() and GetQueueState() provide the
same
> > > information?  
> > 
> > +1
> > 
> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > src/network/model/net-device.h:353: * queue index.
> > On 2011/07/26 14:30:00, Tom Henderson wrote:
> > > Why not return the QueueState also in the callback, to avoid the client
> > needing
> > > to call back into the device with GetQueueState()?
> > 
> > In general, I think that it is better to make such callbacks just query the
> > object that generated the event for anything they need. i.e., in this case,
> what
> > you need is to provide at least one extra argument that is the
Ptr<NetDevice>
> on
> > which the event occured.
> 
> Yesterday I spend much time to understand this review, and finally I have more
> questions than responses :
> 
>   1. the concerned queue are the equivalent of queue disciplines found in the
> Linux Kernel ?
> 
>   2. if YES will NS3 manage this queues under netdevice or elsewhere ?
> 
>   3. concerning DCE kernel usage I see that today we call the
> NS3::NetDevice::Send after the processings of queues so
>      from our point of view the NS3::NetDevice::Send is the equivalent of the
> Kernel  ndo_start_xmit which is low level
>      and do not take care of queues, then it should also return a code like :
> NETDEV_TX_OK or NETDEV_TX_BUSY.
> 
>   4. So if the point 3 is true in DCE / Kernel mode the qdisc will be still
> managed in linux kernel and not in NS3 ?
> 
> Else I would like to have the Ptr<NetDevice> parameter in the Callback setted
by
> SetQueueStateChangeCallback because recently I use AddLinkChangeCallback using
> an annoying callback without parameters.
> 
> Meanwhile I will test the current patch proposition, and I think it will be
OK.

I just read the book 'Understanding Linux Network Internals' and this is what I
understood:

The methods of Linux netdevice dealing about queue are these :

  netif_start_queue: Enables transmission for the device
  netif_stop_queue: Disables transmissions 
  netif_queue_stopped: Returns the status of the egress queue: enabled or
disabled.
  netif_wake_queue: like netif_start_queue but ask the kernel if something is
waiting for transmitting

NOTE: start and stop are only callable by the device drivers.
 
The big difference with the proposed NetDevice patch is that these methods deal
with one and only one queue there is 
no notion of index. I do not understand why we can query a queue state if Send
do not permit to choose the queue ?

Also I reply yes to Tom question:

>Should we provide a separate (Hard)StartXmit() method and document that this
new
>queue flow control API pertains to that method, and not necessarily to
>NetDevice::Send() (which may support qdiscs in the future).  We could document
>that NetDevice::Send() is analogous to dev_queue_xmit() (i.e. it does not
>receive fully formatted frames) but NetDevice::StartXmit() is analogous to
>hard_start_xmit() and accepts frames ready for wire transmission?

Yes. A HardStartXmit with a return code like NETDEV_TX_OK, NETDEV_TX_BUSY (and
for NETDEV_TX_LOCKED I am not sure ! )

So from the kernel I need to call a netif_queue_stopped ( <==> GetQueueState (0)
) to query the device status,
then I would like to be awaken by the callback setted by ( <==>
SetQueueStateChangeCallback ( )) when the NS3::netdevice is ready again to
transmit (NetDevice will call the internal netif_wake_queue for example ).
Finally a  HardStartXmit will be welcome to NS3:Netdevice.

[Ruben Merz, 2011-09-20 19:09:24]

On 2011/07/26 14:30:00, Tom Henderson wrote:
> Is there a plan for allowing qdiscs someday in the NetDevice?  We have
> previously documented that NetDevice::Send() is analogous to dev_queue_xmit()
> but all of these new methods introduced here are really more akin to
> hard_start_xmit() modeling.

I don't have a plan to implement something akin to qdisc right now. But I was
aware of it when working on the current work. So I hope that if someone were to
do it, it could fit in there.

Regarding the hard_start_xmit() modeling, I'm not so sure what you mean.

> Should we provide a separate (Hard)StartXmit() method and document that this
new
> queue flow control API pertains to that method, and not necessarily to
> NetDevice::Send() (which may support qdiscs in the future).  We could document
> that NetDevice::Send() is analogous to dev_queue_xmit() (i.e. it does not
> receive fully formatted frames) but NetDevice::StartXmit() is analogous to
> hard_start_xmit() and accepts frames ready for wire transmission?


> http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
> File src/network/model/net-device.h (right):
> 
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> src/network/model/net-device.h:339: virtual QueueState GetQueueState (uint32_t
> index) = 0;
> I think it would help to explain a bit in the doxygen what this index is.  I
> think the answer is that there may be multiple queues, indexed in a way that
is
> specific to the net device, and even though the NetDevice API doesn't allow
the
> user to specify which queue the packet will get filtered into, if the upper
> layer somehow knows this information, it can discern what the possible fate of
a
> packet would be, if it were to be sent into the NetDevice.  Is this correct?

Agreed.
 
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> src/network/model/net-device.h:347: */
> Why is this needed?  Can't GetNQueues() and GetQueueState() provide the same
> information?

GetNQueues tells you how many queues the underlying netdevice has. GetQueueState
tells you, for a particular queue, what is current state is.

What would be your suggestion to combine both?
 
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> src/network/model/net-device.h:353: * queue index.
> Why not return the QueueState also in the callback, to avoid the client
needing
> to call back into the device with GetQueueState()?

Because you may have multiple queues in a netdevice, where some might be full
and some not. Then, I believe on the netdevice knows whether it can accept more
packets or not. It should not be up to the upper layer to decide this.

Ruben

[Ruben Merz, 2011-09-20 21:23:25]

On 2011/09/14 06:47:43, Mathieu Lacage wrote:
> I am sorry if this has been discussed before but I reviewed this carefully
based
> on the latest kernel headers. The biggest differences I see here are:
>    1) the Linux kernel allows the driver to decide which queue to use to
output
> a specific packet _but_ it is allowed to communicate this information to the
> higher level layers. i.e., PickTxQueue would take a Ptr<Packet> and return a
> queue index.
>    2) the linux kernel allows the higher level layers to pick which queue to
use
> for a specific packet. They store this in one of the skb fields.
> 
> So, what is the impact from the perspective of integrating a linux kernel ip
> stack on top of this ? I think that it is pretty much nil because the only
user
> of this API in the linux kernel is the 802.11 stack (the packet scheduling API
> has a similarly-named function select_queue but it uses to manage the
scheduler
> queues, not the device queues).
> 
> So, I think that the above is good. It would be really nice if frederic could
> try it out to make sure it is ok because this has a fairly large API impact.
> 
> http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
> File src/network/model/net-device.h (right):
> 
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> src/network/model/net-device.h:347: */
> On 2011/07/26 14:30:00, Tom Henderson wrote:
> > Why is this needed?  Can't GetNQueues() and GetQueueState() provide the same
> > information?  
> 
> +1

Please see my earlier answer to Tom.

>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> src/network/model/net-device.h:353: * queue index.
> On 2011/07/26 14:30:00, Tom Henderson wrote:
> > Why not return the QueueState also in the callback, to avoid the client
> needing
> > to call back into the device with GetQueueState()?
> 
> In general, I think that it is better to make such callbacks just query the
> object that generated the event for anything they need. i.e., in this case,
what
> you need is to provide at least one extra argument that is the Ptr<NetDevice>
on
> which the event occured.

I'm afraid I don't really understand what you are getting at.

[Ruben Merz, 2011-09-20 21:37:12]

[snip]

> The big difference with the proposed NetDevice patch is that these methods
deal
> with one and only one queue there is 
> no notion of index. I do not understand why we can query a queue state if Send
> do not permit to choose the queue ?

Good point. It was a requirement that came out of earlier discussions. In light
of the possibility of implementing a HardStartXmit(), it might not be necessary
anymore.

What is still necessary is something that tells the upper layer whether the
netdevice can accept any more packet (which is actually the original reason for
this patchset).

> Also I reply yes to Tom question:
> 
> >Should we provide a separate (Hard)StartXmit() method and document that this
> new
> >queue flow control API pertains to that method, and not necessarily to
> >NetDevice::Send() (which may support qdiscs in the future).  We could
document
> >that NetDevice::Send() is analogous to dev_queue_xmit() (i.e. it does not
> >receive fully formatted frames) but NetDevice::StartXmit() is analogous to
> >hard_start_xmit() and accepts frames ready for wire transmission?
> 
> Yes. A HardStartXmit with a return code like NETDEV_TX_OK, NETDEV_TX_BUSY (and
> for NETDEV_TX_LOCKED I am not sure ! )

Thanks to Frederic's explanation, I understand better the reason behind having a
HardStartXmit. Could be a good idea, indeed. But, one thing I need to emphasize
is that some way for layers above the netdevice to stop sending if the queues
are full is still required.

Ruben
 
> So from the kernel I need to call a netif_queue_stopped ( <==> GetQueueState
(0)
> ) to query the device status,
> then I would like to be awaken by the callback setted by ( <==>
> SetQueueStateChangeCallback ( )) when the NS3::netdevice is ready again to
> transmit (NetDevice will call the internal netif_wake_queue for example ).
> Finally a  HardStartXmit will be welcome to NS3:Netdevice.

[Tom Henderson, 2011-09-21 13:44:12]

> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > src/network/model/net-device.h:347: */
> > Why is this needed?  Can't GetNQueues() and GetQueueState() provide the same
> > information?
> 
> GetNQueues tells you how many queues the underlying netdevice has.
GetQueueState
> tells you, for a particular queue, what is current state is.
> 
> What would be your suggestion to combine both?

the comment was directed at the method IsTxPathBlocked, which is implemented (in
WifiNetDevice) by the other two methods GetNQueues and GetQueueState.  

Are there other conditions besides all queues being in stop state that are
expected for this method (for other net devices)?
>  
> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > src/network/model/net-device.h:353: * queue index.
> > Why not return the QueueState also in the callback, to avoid the client
> needing
> > to call back into the device with GetQueueState()?
> 
> Because you may have multiple queues in a netdevice, where some might be full
> and some not. Then, I believe on the netdevice knows whether it can accept
more
> packets or not. It should not be up to the upper layer to decide this.
> 

My comment was just that if you are notifying a handler that a queue state
changed and you are providing its index, you may as well provide it with the
state change information.  For example, WifiPhyStateHelper callback
m_stateLogger provides the state information within the callback.

[Mathieu Lacage, 2012-02-24 08:27:24]

On 2011/09/20 21:23:25, Ruben Merz wrote:
> On 2011/09/14 06:47:43, Mathieu Lacage wrote:
> > I am sorry if this has been discussed before but I reviewed this carefully
> based
> > on the latest kernel headers. The biggest differences I see here are:
> >    1) the Linux kernel allows the driver to decide which queue to use to
> output
> > a specific packet _but_ it is allowed to communicate this information to the
> > higher level layers. i.e., PickTxQueue would take a Ptr<Packet> and return a
> > queue index.
> >    2) the linux kernel allows the higher level layers to pick which queue to
> use
> > for a specific packet. They store this in one of the skb fields.
> > 
> > So, what is the impact from the perspective of integrating a linux kernel ip
> > stack on top of this ? I think that it is pretty much nil because the only
> user
> > of this API in the linux kernel is the 802.11 stack (the packet scheduling
API
> > has a similarly-named function select_queue but it uses to manage the
> scheduler
> > queues, not the device queues).
> > 
> > So, I think that the above is good. It would be really nice if frederic
could
> > try it out to make sure it is ok because this has a fairly large API impact.
> > 
> >
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
> > File src/network/model/net-device.h (right):
> > 
> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > src/network/model/net-device.h:347: */
> > On 2011/07/26 14:30:00, Tom Henderson wrote:
> > > Why is this needed?  Can't GetNQueues() and GetQueueState() provide the
same
> > > information?  
> > 
> > +1
> 
> Please see my earlier answer to Tom.
> 
> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > src/network/model/net-device.h:353: * queue index.
> > On 2011/07/26 14:30:00, Tom Henderson wrote:
> > > Why not return the QueueState also in the callback, to avoid the client
> > needing
> > > to call back into the device with GetQueueState()?
> > 
> > In general, I think that it is better to make such callbacks just query the
> > object that generated the event for anything they need. i.e., in this case,
> what
> > you need is to provide at least one extra argument that is the
Ptr<NetDevice>
> on
> > which the event occured.
> 
> I'm afraid I don't really understand what you are getting at.

The callback needs an extra argument, the Ptr<NetDevice> itself. The callback
can then call GetQueueState if needed. In general, it is better to make
notification listeners not be notified of the exact kind of state change. It is
better to just notify them that the state changed and that they need to adjust
their behavior accordingly. Two reasons for this:
  - This makes the notification code easier to write.
  - the listening code in general is too complicated to write to handle
incremental changes to the state. Most serious listeners never try to calculate
the delta. They just calculate the new state.

[ruben.merz, 2012-02-24 11:17:54]

On 2012/02/24 08:27:24, Mathieu Lacage wrote:
> On 2011/09/20 21:23:25, Ruben Merz wrote:
> > On 2011/09/14 06:47:43, Mathieu Lacage wrote:
> > > I am sorry if this has been discussed before but I reviewed this carefully
> > based
> > > on the latest kernel headers. The biggest differences I see here are:
> > >    1) the Linux kernel allows the driver to decide which queue to use to
> > output
> > > a specific packet _but_ it is allowed to communicate this information to
the
> > > higher level layers. i.e., PickTxQueue would take a Ptr<Packet> and return
a
> > > queue index.
> > >    2) the linux kernel allows the higher level layers to pick which queue
to
> > use
> > > for a specific packet. They store this in one of the skb fields.
> > > 
> > > So, what is the impact from the perspective of integrating a linux kernel
ip
> > > stack on top of this ? I think that it is pretty much nil because the only
> > user
> > > of this API in the linux kernel is the 802.11 stack (the packet scheduling
> API
> > > has a similarly-named function select_queue but it uses to manage the
> > scheduler
> > > queues, not the device queues).
> > > 
> > > So, I think that the above is good. It would be really nice if frederic
> could
> > > try it out to make sure it is ok because this has a fairly large API
impact.
> > > 
> > >
> http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device.h
> > > File src/network/model/net-device.h (right):
> > > 
> > >
> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > > src/network/model/net-device.h:347: */
> > > On 2011/07/26 14:30:00, Tom Henderson wrote:
> > > > Why is this needed?  Can't GetNQueues() and GetQueueState() provide the
> same
> > > > information?  
> > > 
> > > +1
> > 
> > Please see my earlier answer to Tom.
> > 
> > >
> >
>
http://codereview.appspot.com/4664057/diff/2001/src/network/model/net-device....
> > > src/network/model/net-device.h:353: * queue index.
> > > On 2011/07/26 14:30:00, Tom Henderson wrote:
> > > > Why not return the QueueState also in the callback, to avoid the client
> > > needing
> > > > to call back into the device with GetQueueState()?
> > > 
> > > In general, I think that it is better to make such callbacks just query
the
> > > object that generated the event for anything they need. i.e., in this
case,
> > what
> > > you need is to provide at least one extra argument that is the
> Ptr<NetDevice>
> > on
> > > which the event occured.
> > 
> > I'm afraid I don't really understand what you are getting at.
> 
> The callback needs an extra argument, the Ptr<NetDevice> itself. The callback
> can then call GetQueueState if needed. In general, it is better to make
> notification listeners not be notified of the exact kind of state change. It
is
> better to just notify them that the state changed and that they need to adjust
> their behavior accordingly. Two reasons for this:
>   - This makes the notification code easier to write.
>   - the listening code in general is too complicated to write to handle
> incremental changes to the state. Most serious listeners never try to
calculate
> the delta. They just calculate the new state.

I see, do you know a good example in the ns-3 code base?